mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
fix(vm-disk, kubernetes-nodes)!: refuse golden images whose import keeps failing (#4802)
## What this PR does New clones can wait forever on a golden image whose import keeps failing. CDI v1.66.1 retries ordinary HTTP failures without setting phase `Failed`. The guards now also refuse goldens with at least three importer restarts unless they reached `Succeeded`. kubernetes-nodes and the picker require `Running=False`; vm-disk refuses during running attempts too. Existing vm-disks and already-cloned pools remain exempt. CDI never lowers the restart count, so a healthy import with three early restarts stays refusable until `Succeeded`, including while a recreated importer is still starting. An empty worker-pool StorageClass now resolves to the virtualization default, then the GA Kubernetes default. Beta is a last fallback for the class written explicitly onto a worker disk, not a CDI fallback. The strict same-class policy stays. This can refuse a previously accepted builtin pool on upgrade. The affected pool has an empty StorageClass, no cloned disk yet, and a golden on the Kubernetes default while another class is the virtualization default. StorageClass is immutable: point `osImage.builtin` at a catalog entry on the virtualization default or switch to `osImage.factory`. Already-cloned pools keep their disk's class. A VMDisk's own HTTP import, golden-image health checks and imports that hang without restarting remain outside this change. Closes #4564 Refs #4059 ### Screenshots No UI changes. ### Downstream repositories The platform values change only updates a comment. The application schema change updates the StorageClass description without changing fields, defaults or enums. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(vm-disk, kubernetes-nodes)!: refuse new clones of golden images whose import has not succeeded after at least three importer restarts. kubernetes-nodes and the image picker require the importer to be not running; vm-disk refuses during running attempts too. kubernetes-nodes now resolves an empty storageClass to CDI's virtualization default before the Kubernetes default. A builtin pool with an empty storageClass, no cloned disk yet, and a golden on a different class can now be refused. Point it at a catalog entry on the virtualization default or switch to osImage.factory. Already-cloned pools are unaffected. ```
This commit is contained in:
25 files changed
+809
-87
No files matched your search
+1
-1
@@ -22,7 +22,7 @@ type ConfigSpec struct {
|
||||
// +kubebuilder:default:=""
|
||||
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="cluster is immutable"
|
||||
Cluster string `json:"cluster"`
|
||||
// StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`.
|
||||
// StorageClass for the worker node system disk. When empty, CDI picks the class: the one annotated `storageclass.kubevirt.io/is-default-virt-class`, else the cluster default. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: the chart refuses a pool whose class differs from its golden's, because across classes CDI either falls back to a host-assisted copy over the pod network (different CSI drivers or volume modes) or leaves a clone into the other class to the CSI driver. Leaving it empty makes the same comparison against the class CDI would pick. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`.
|
||||
// +kubebuilder:default:="replicated"
|
||||
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="storageClass is immutable"
|
||||
StorageClass string `json:"storageClass"`
|
||||
|
||||
@@ -13,10 +13,10 @@
|
||||
# silently breaks every fresh install using osImage.builtin, and nothing catches it
|
||||
# until a ~95-minute e2e run does.
|
||||
#
|
||||
# The StorageClass is the same trap from the other direction: CDI cannot
|
||||
# CSI-clone across StorageClasses (it would silently fall back to a host-assisted
|
||||
# copy over the pod network), so the render also rejects a group whose
|
||||
# storageClass differs from its golden's. Bumping one file's default alone brings
|
||||
# The StorageClass is the same trap from the other direction: the render also
|
||||
# rejects a group whose storageClass differs from its golden's, because across
|
||||
# classes CDI either falls back to a host-assisted copy over the pod network or
|
||||
# leaves the clone to the CSI driver. Bumping one file's default alone brings
|
||||
# that guard down on the default path.
|
||||
#
|
||||
# Both checks are cheap and exact, which is the point — they turn a slow, remote
|
||||
@@ -70,8 +70,8 @@ CATALOG_VALUES="$REPO_ROOT/packages/system/kubernetes-worker-image/values.yaml"
|
||||
echo "StorageClass mismatch on the default osImage.builtin path:" >&2
|
||||
echo " packages/apps/kubernetes-nodes/values.yaml storageClass: $app_sc" >&2
|
||||
echo " kubernetes-worker-image golden effective storageClass: $entry_sc" >&2
|
||||
echo "CDI cannot CSI-clone across StorageClasses, so the tenant render" >&2
|
||||
echo "rejects this outright. Keep the two defaults in step." >&2
|
||||
echo "The tenant render refuses a pool whose storageClass differs from" >&2
|
||||
echo "its golden's. Keep the two defaults in step." >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
@@ -91,6 +91,30 @@ spec:
|
||||
(conditions[?type == 'WorkloadsReady']):
|
||||
- status: "False"
|
||||
reason: DataVolumeNotReady
|
||||
# The golden image guard in the kubernetes-nodes chart and the image option
|
||||
# list call an import failing once its importer has restarted three times and
|
||||
# is not running, and the vm-disk guard once it has restarted three times,
|
||||
# because CDI keeps an unreachable URL in progress instead of moving it to
|
||||
# Failed. This pins that CDI does report both. The default kubelet back-off
|
||||
# before the third restart adds up to 70s (10s, 20s, 40s), and the timeout
|
||||
# leaves room for the importer's own runs on top of that. The count is
|
||||
# compared as a string, because Chainsaw v0.2.15 mis-evaluates numeric
|
||||
# literals; 'null' in the list makes an absent count, which CDI omits at
|
||||
# zero, fail too.
|
||||
- name: wait-importer-restarts
|
||||
try:
|
||||
- assert:
|
||||
timeout: 4m
|
||||
resource:
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
kind: DataVolume
|
||||
metadata:
|
||||
name: vm-disk-unreachable
|
||||
status:
|
||||
phase: ImportInProgress
|
||||
(contains(['null', '0', '1', '2'], to_string(restartCount))): false
|
||||
(conditions[?type == 'Running']):
|
||||
- status: "False"
|
||||
---
|
||||
# VMDisk clone paths: source.image clones a golden image from cozy-public, and
|
||||
# source.disk clones that disk in turn, so only the golden image downloads.
|
||||
|
||||
@@ -23,9 +23,9 @@ The pool's Kubernetes minor (`version`) may lag the cluster's during an upgrade
|
||||
|
||||
### Common Parameters
|
||||
|
||||
| Name | Description | Type | Value |
|
||||
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------ |
|
||||
| `storageClass` | StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`. | `string` | `replicated` |
|
||||
| Name | Description | Type | Value |
|
||||
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------ |
|
||||
| `storageClass` | StorageClass for the worker node system disk. When empty, CDI picks the class: the one annotated `storageclass.kubevirt.io/is-default-virt-class`, else the cluster default. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: the chart refuses a pool whose class differs from its golden's, because across classes CDI either falls back to a host-assisted copy over the pod network (different CSI drivers or volume modes) or leaves a clone into the other class to the CSI driver. Leaving it empty makes the same comparison against the class CDI would pick. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`. | `string` | `replicated` |
|
||||
|
||||
|
||||
### Pool Sizing
|
||||
|
||||
@@ -441,18 +441,25 @@ still fails.
|
||||
{{- end }}
|
||||
|
||||
{{- /*
|
||||
Name of the cluster's default StorageClass, or the empty string when there is
|
||||
none (and always under `helm template`, which has no cluster to read).
|
||||
Name of the StorageClass CDI binds a kubevirt-content DataVolume on when it
|
||||
names no class, or the empty string when there is none (and always under `helm
|
||||
template`, which has no cluster to read). CDI takes a class annotated
|
||||
storageclass.kubevirt.io/is-default-virt-class first and only then the
|
||||
Kubernetes default (GetStorageClassByNameWithVirtFallback in CDI's
|
||||
pkg/controller/common/util.go), and both the goldens and the worker disks are
|
||||
kubevirt content.
|
||||
|
||||
An empty storageClass is a documented, schema-valid setting on both a worker pool
|
||||
and a worker image catalog entry, and it means "the cluster default". Without
|
||||
resolving it the golden-versus-pool StorageClass comparison simply skips whenever
|
||||
either side is empty, which is the one corner where skipping is worst: CDI then
|
||||
falls back to a host-assisted copy over the pod network, silently, and that copy
|
||||
is the transfer the clone path exists to remove.
|
||||
either side is empty, and lets a cross-class clone through unchecked.
|
||||
|
||||
Both the current annotation and its beta predecessor count, because clusters
|
||||
provisioned years apart carry different ones and Kubernetes still honours both.
|
||||
The beta predecessor of the Kubernetes annotation is read last, only when no
|
||||
class carries the current one. CDI does not read it: with neither of the other
|
||||
two defaults it creates no claim at all for a DataVolume that names no class
|
||||
and no access modes (pvcFromStorage, ErrStorageClassNotFound). Read here, it
|
||||
still gives a pool on such a cluster a class, written onto the worker disk
|
||||
explicitly.
|
||||
|
||||
More than one class may carry the annotation at once. Kubernetes permits that --
|
||||
it is the normal state midway through swapping a cluster's default -- and
|
||||
@@ -463,16 +470,21 @@ that is actually in force. Timestamps are RFC3339 in UTC, so comparing them as
|
||||
strings is comparing them chronologically.
|
||||
*/ -}}
|
||||
{{- define "kubernetes-nodes.defaultStorageClassName" -}}
|
||||
{{- $classes := lookup "storage.k8s.io/v1" "StorageClass" "" "" -}}
|
||||
{{- $classes := dig "items" (list) ((lookup "storage.k8s.io/v1" "StorageClass" "" "") | default dict) -}}
|
||||
{{- $name := "" -}}
|
||||
{{- $createdAt := "" -}}
|
||||
{{- range (dig "items" (list) ($classes | default dict)) -}}
|
||||
{{- $annotations := dig "metadata" "annotations" (dict) . -}}
|
||||
{{- if or (eq (dig "storageclass.kubernetes.io/is-default-class" "" $annotations | toString) "true") (eq (dig "storageclass.beta.kubernetes.io/is-default-class" "" $annotations | toString) "true") -}}
|
||||
{{- $at := dig "metadata" "creationTimestamp" "" . | toString -}}
|
||||
{{- if or (not $name) (gt $at $createdAt) -}}
|
||||
{{- $name = dig "metadata" "name" "" . | toString -}}
|
||||
{{- $createdAt = $at -}}
|
||||
{{- range list "storageclass.kubevirt.io/is-default-virt-class" "storageclass.kubernetes.io/is-default-class" "storageclass.beta.kubernetes.io/is-default-class" -}}
|
||||
{{- $key := . -}}
|
||||
{{- $createdAt := "" -}}
|
||||
{{- if not $name -}}
|
||||
{{- range $classes -}}
|
||||
{{- $annotations := dig "metadata" "annotations" (dict) . -}}
|
||||
{{- if eq (dig $key "" $annotations | toString) "true" -}}
|
||||
{{- $at := dig "metadata" "creationTimestamp" "" . | toString -}}
|
||||
{{- if or (not $name) (gt $at $createdAt) -}}
|
||||
{{- $name = dig "metadata" "name" "" . | toString -}}
|
||||
{{- $createdAt = $at -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
@@ -269,7 +269,7 @@ spec:
|
||||
{{- $httpURL := printf "%s/image/%s/%s/openstack-amd64.raw.xz" ($reqFactoryURL | trimSuffix "/") $reqSchematicID $reqVersion }}
|
||||
{{- $goldenName := printf "talos-worker-%s-%s" $reqSchematicID $reqVersion }}
|
||||
{{- /* A builtin pool whose golden is absent, failed to import, or on a
|
||||
StorageClass the worker disk cannot CSI-clone from fails loudly here
|
||||
StorageClass other than the worker disk's fails loudly here
|
||||
rather than degrading silently. And because a CDI clone target must be
|
||||
at least as large as its source, a worker diskSize below the golden's
|
||||
storage would fail the clone at runtime — reject it at render when the
|
||||
@@ -302,7 +302,7 @@ spec:
|
||||
and no CR edit is needed to trigger the abort, only the next periodic
|
||||
reconcile. The comment below rejects exactly that blast radius for a
|
||||
mid-import golden; applying the same reasoning to a golden that is
|
||||
absent or Failed is what it costs to be consistent.
|
||||
absent or failing to import is what it costs to be consistent.
|
||||
|
||||
So the loud failure stays where it is the only signal — a pool still
|
||||
waiting for its first disk — and a pool that is past that point is
|
||||
@@ -334,7 +334,7 @@ spec:
|
||||
from it too, and the two disagreeing is how a pool at storageClass ""
|
||||
passed the comparison on the class its existing disks are bound to and
|
||||
then bound its next disk on whatever the cluster default had become by
|
||||
then -- taking the host-assisted copy the guard exists to refuse, with
|
||||
then -- taking the cross-class clone the guard exists to refuse, with
|
||||
the guard reporting no drift.
|
||||
|
||||
A pool that declares no class took the default in force when ITS disks
|
||||
@@ -367,35 +367,51 @@ spec:
|
||||
{{- fail (printf "nodeGroup %q: osImage.builtin requested golden image %q not found in cozy-public, neither as a DataVolume nor as a PVC — enable the cozystack.kubernetes-worker-image catalog via bundles.enabledPackages with a matching (schematicID, version) entry, or switch this pool to osImage.factory." .groupName $goldenName) }}
|
||||
{{- end }}
|
||||
{{- else }}
|
||||
{{- /* Fail only on the golden's terminal failure phase, not on "not yet
|
||||
{{- /* Fail only on a golden whose import is failing, not on "not yet
|
||||
Succeeded". A golden mid-import is transient: CDI holds the worker
|
||||
DataVolume until the source is populated, which is the correct
|
||||
behaviour and recovers unattended. Hard-failing the render there would
|
||||
abort the ENTIRE KubernetesNodes HelmRelease — every resource, not just
|
||||
this disk — for every pool pinned to that golden, blocking unrelated
|
||||
scale operations for the length of an import. Only Failed, which
|
||||
never recovers without operator action, justifies that blast radius.
|
||||
scale operations for the length of an import. Failed never recovers
|
||||
without operator action, but CDI reaches it only on a fatal error such
|
||||
as a checksum mismatch: an unreachable URL restarts the importer with
|
||||
back-off while the phase stays in progress. So an importer that
|
||||
restarted at least three times and is not running now is refused too;
|
||||
that refusal lifts on its own once an attempt succeeds.
|
||||
Falling back to the HTTP import is deliberately not an option either: it
|
||||
would make disk-system's source depend on cluster state, so the source —
|
||||
and with it the content-hashed KubevirtMachineTemplate name — would flip
|
||||
once the golden went Ready and roll every worker in the pool. */}}
|
||||
{{- $goldenPhase := dig "status" "phase" "" $goldenDV | toString }}
|
||||
{{- $goldenRestarts := dig "status" "restartCount" 0 $goldenDV | int }}
|
||||
{{- $goldenRunning := "" }}
|
||||
{{- range (dig "status" "conditions" (list) $goldenDV) }}
|
||||
{{- if eq (dig "type" "" . | toString) "Running" }}{{- $goldenRunning = dig "status" "" . | toString }}{{- end }}
|
||||
{{- end }}
|
||||
{{- if and (eq $goldenPhase "Failed") (not $poolCloned) }}
|
||||
{{- fail (printf "nodeGroup %q: golden DataVolume %q in cozy-public failed to import (phase Failed) — cloning it would block this pool's workers indefinitely. Check the importer pod in cozy-public, then delete the DataVolume to retry the import." .groupName $goldenName) }}
|
||||
{{- else if and (ne $goldenPhase "Succeeded") (ge $goldenRestarts 3) (eq $goldenRunning "False") (not $poolCloned) }}
|
||||
{{- fail (printf "nodeGroup %q: golden DataVolume %q in cozy-public keeps failing to import (%d importer restarts) — cloning it would block this pool's workers while it does. The refusal lifts by itself while a new import attempt is running, and for good once one succeeds. The importer is retried with a growing back-off, and a recreated importer pod counts as not running until its container starts, so after the source is fixed the refusal lasts at least until the current back-off delay ends and the next importer starts. Check the importer pod logs in cozy-public." .groupName $goldenName $goldenRestarts) }}
|
||||
{{- end }}
|
||||
{{- /* CDI takes the storage-layer CSI-clone path only when source and target
|
||||
share a StorageClass; across classes it silently falls back to a
|
||||
{{- /* Keep a pool on its golden's StorageClass. CDI falls back to a
|
||||
host-assisted copy over the pod network — exactly the per-worker
|
||||
network transfer osImage.builtin exists to avoid (#3231). Reject the
|
||||
case we can prove. An empty class on either side means "the cluster
|
||||
default", which is resolvable against a live cluster and is resolved
|
||||
here: leaving it unresolved is what let the one corner that CANNOT be
|
||||
a clone — a pool at storageClass "" whose golden is on an explicit
|
||||
class — pass unchecked. With no cluster to read (helm template) both
|
||||
stay empty and the comparison is skipped, as before. This rules out
|
||||
the guaranteed-host-assisted case, not every one: a matching
|
||||
StorageClass still leaves the strategy to the CDI StorageProfile,
|
||||
which the worker DataVolume inherits. */}}
|
||||
network transfer osImage.builtin exists to avoid (#3231) — when the
|
||||
two claims have different CSI drivers or volume modes
|
||||
(validateAdvancedClonePVC in CDI's pkg/controller/clone/planner.go).
|
||||
Two classes of one driver do not fall back: CDI asks that driver for
|
||||
a snapshot or CSI clone into the other class, and whether that works
|
||||
depends on the driver and on both classes' parameters. Sharing a
|
||||
class rules out both a driver mismatch and a cross-class clone. An
|
||||
empty class on either side means the class CDI binds on, which is
|
||||
resolvable against a live cluster and is resolved here: leaving it
|
||||
unresolved is what let a pool at storageClass "" whose golden is on
|
||||
an explicit class pass unchecked. With no cluster to read (helm template) both stay empty
|
||||
and the comparison is skipped, as before. A matching StorageClass
|
||||
does not rule out every host-assisted copy: the strategy still comes
|
||||
from the CDI StorageProfile the worker DataVolume inherits, and CDI
|
||||
also falls back on differing volume modes, when no VolumeSnapshotClass
|
||||
fits, or when a larger target cannot be expanded. */}}
|
||||
{{- $workerSC := $cloneWorkerSC }}
|
||||
{{- $goldenSC := dig "spec" "storage" "storageClassName" "" $goldenDV | toString }}
|
||||
{{- /* A golden that declares no class is not on "the cluster default" as it
|
||||
@@ -427,9 +443,9 @@ spec:
|
||||
made, and reports it as the empty string. */}}
|
||||
{{- $defaultNote := "" }}
|
||||
{{- if $resolvedFromDefault }}
|
||||
{{- $defaultNote = printf " An empty storageClass was resolved to the cluster default StorageClass %q." $defaultSC }}
|
||||
{{- $defaultNote = printf " An empty storageClass was resolved to %q, the class CDI binds a disk that names none on." $defaultSC }}
|
||||
{{- end }}
|
||||
{{- fail (printf "nodeGroup %q: storageClass %q differs from golden image %q storageClass %q — CDI cannot CSI-clone across StorageClasses and would fall back to a host-assisted copy over the pod network.%s Neither side can be moved in place: this golden's class is fixed for its lifetime, because a DataVolume spec cannot be patched, and the pool's storageClass is immutable on an existing pool. So add a NEW kubernetes-worker-image catalog entry with a different (schematicID, version) on storageClass %q (kubernetesWorkerImage.images in the cozystack-platform Package values) and point this pool's osImage.builtin at it; or, once no pool clones this golden, delete it and let the catalog recreate it on that class. Switching this pool to osImage.factory also resolves it." .groupName $workerSC $goldenName $goldenSC $defaultNote $workerSC) }}
|
||||
{{- fail (printf "nodeGroup %q: storageClass %q differs from golden image %q storageClass %q — across StorageClasses CDI either falls back to a host-assisted copy over the pod network (different CSI drivers or volume modes) or leaves a clone into the other class to the CSI driver.%s Neither side can be moved in place: this golden's class is fixed for its lifetime, because a DataVolume spec cannot be patched, and the pool's storageClass is immutable on an existing pool. So add a NEW kubernetes-worker-image catalog entry with a different (schematicID, version) on storageClass %q (kubernetesWorkerImage.images in the cozystack-platform Package values) and point this pool's osImage.builtin at it; or, once no pool clones this golden, delete it and let the catalog recreate it on that class. Switching this pool to osImage.factory also resolves it." .groupName $workerSC $goldenName $goldenSC $defaultNote $workerSC) }}
|
||||
{{- end }}
|
||||
{{- $goldenStorage := dig "spec" "storage" "resources" "requests" "storage" "" $goldenDV }}
|
||||
{{- /* Same authority argument as the class one line up: a golden whose spec
|
||||
@@ -529,14 +545,15 @@ spec:
|
||||
rejects RWX volumes that are not on a DRBD-backed StorageClass,
|
||||
which is why the pool's storageClass defaults to a replicated/DRBD
|
||||
class rather than to the cluster default. An empty storageClass
|
||||
omits the field so the cluster default applies.
|
||||
omits the field, and CDI binds the disk on its virtualization
|
||||
default or, without one, the Kubernetes default.
|
||||
|
||||
A builtin pool instead renders the class the guard above resolved for
|
||||
it. Omitting the field there would let the disk bind on whatever the
|
||||
cluster default is at bind time, and CDI takes the storage-layer path
|
||||
only when source and target share a class -- so a default that moved
|
||||
after the pool's earlier disks were bound turns the next clone into a
|
||||
host-assisted copy over the pod network, the transfer the guard exists
|
||||
cluster default is at bind time -- so a default that moved after the
|
||||
pool's earlier disks were bound turns the next clone into a
|
||||
cross-class one, which CDI either runs as a host-assisted copy over
|
||||
the pod network or leaves to the CSI driver, the clone the guard exists
|
||||
to refuse, while the guard reports no drift because it compared the
|
||||
class those earlier disks sit on. With the class empty and no cluster
|
||||
to resolve against the field is still omitted, so the HTTP path does
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
suite: osImage.builtin default-StorageClass selection with only the beta annotation
|
||||
|
||||
# CDI does not read the beta annotation, so the pool's class comes from the
|
||||
# chart alone, written onto the worker disk explicitly.
|
||||
|
||||
templates:
|
||||
- templates/nodegroup.yaml
|
||||
|
||||
release:
|
||||
name: kubernetes-nodes-myk8s-md0
|
||||
namespace: tenant-test
|
||||
|
||||
set:
|
||||
cluster: myk8s
|
||||
_cluster:
|
||||
cluster-domain: cozy.local
|
||||
version: "v1.35"
|
||||
minReplicas: 1
|
||||
maxReplicas: 3
|
||||
instanceType: u1.medium
|
||||
diskSize: 20Gi
|
||||
storageClass: ""
|
||||
roles: [ingress-nginx]
|
||||
osImage:
|
||||
builtin: {}
|
||||
|
||||
kubernetesProvider:
|
||||
scheme:
|
||||
"instancetype.kubevirt.io/v1beta1/VirtualMachineClusterInstancetype":
|
||||
gvr: {group: instancetype.kubevirt.io, version: v1beta1, resource: virtualmachineclusterinstancetypes}
|
||||
namespaced: false
|
||||
"cluster.x-k8s.io/v1beta1/MachineSet":
|
||||
gvr: {group: cluster.x-k8s.io, version: v1beta1, resource: machinesets}
|
||||
namespaced: true
|
||||
"infrastructure.cluster.x-k8s.io/v1alpha1/KubevirtMachineTemplate":
|
||||
gvr: {group: infrastructure.cluster.x-k8s.io, version: v1alpha1, resource: kubevirtmachinetemplates}
|
||||
namespaced: true
|
||||
"cdi.kubevirt.io/v1beta1/DataVolume":
|
||||
gvr: {group: cdi.kubevirt.io, version: v1beta1, resource: datavolumes}
|
||||
namespaced: true
|
||||
"storage.k8s.io/v1/StorageClass":
|
||||
gvr: {group: storage.k8s.io, version: v1, resource: storageclasses}
|
||||
namespaced: false
|
||||
objects:
|
||||
- kind: VirtualMachineClusterInstancetype
|
||||
apiVersion: instancetype.kubevirt.io/v1beta1
|
||||
metadata: {name: u1.medium}
|
||||
- kind: MachineSet
|
||||
apiVersion: cluster.x-k8s.io/v1beta1
|
||||
metadata: {name: ignored-ms, namespace: tenant-test}
|
||||
spec: {template: {spec: {infrastructureRef: {kind: Other, name: ignored}}}}
|
||||
- kind: KubevirtMachineTemplate
|
||||
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
|
||||
metadata: {name: ignored-kmt, namespace: tenant-test}
|
||||
# The outgoing default, created first.
|
||||
- kind: StorageClass
|
||||
apiVersion: storage.k8s.io/v1
|
||||
metadata:
|
||||
name: beta-only
|
||||
creationTimestamp: "2026-01-01T00:00:00Z"
|
||||
annotations:
|
||||
storageclass.beta.kubernetes.io/is-default-class: "true"
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata:
|
||||
name: talos-worker-ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515-v1.13.6
|
||||
namespace: cozy-public
|
||||
spec:
|
||||
storage:
|
||||
resources: {requests: {storage: 6Gi}}
|
||||
storageClassName: beta-only
|
||||
status: {phase: Succeeded}
|
||||
|
||||
tests:
|
||||
- it: resolves an empty pool storageClass to a class with only the beta annotation
|
||||
asserts:
|
||||
- notFailedTemplate: {}
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.dataVolumeTemplates[0].spec.storage.storageClassName
|
||||
value: beta-only
|
||||
documentSelector: {path: kind, value: KubevirtMachineTemplate}
|
||||
@@ -1,6 +1,6 @@
|
||||
suite: osImage.builtin golden guards on a pool that has already cloned
|
||||
|
||||
# The absence and Failed guards abort the ENTIRE KubernetesNodes HelmRelease, and
|
||||
# The absence and failed-import guards abort the ENTIRE KubernetesNodes HelmRelease, and
|
||||
# for a pool whose workers all cloned long ago that price buys nothing: replica
|
||||
# changes, kubelet retuning and MachineHealthCheck timeouts have no stake in the
|
||||
# golden, and no CR edit is needed to trigger the abort — the next periodic
|
||||
@@ -152,6 +152,30 @@ kubernetesProvider:
|
||||
resources: {requests: {storage: 6Gi}}
|
||||
storageClassName: replicated
|
||||
status: {phase: Failed}
|
||||
# md0's own disk against a golden whose re-import is now stuck: the pool
|
||||
# already holds its copy, so it needs nothing from the source.
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata:
|
||||
name: disk-system-kubernetes-myk8s-md0-pqrst
|
||||
namespace: tenant-test
|
||||
annotations:
|
||||
kubernetes-worker-image.cozystack.io/golden: talos-worker-stucklater-v1.0.0
|
||||
kubernetes-worker-image.cozystack.io/pool: kubernetes-myk8s-md0
|
||||
status: {phase: Succeeded}
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata: {name: talos-worker-stucklater-v1.0.0, namespace: cozy-public}
|
||||
spec:
|
||||
storage:
|
||||
resources: {requests: {storage: 6Gi}}
|
||||
storageClassName: replicated
|
||||
status:
|
||||
phase: ImportInProgress
|
||||
# A string: helm-unittest's lookup cannot deep-copy an int, and | int takes either.
|
||||
restartCount: "7"
|
||||
conditions:
|
||||
- {type: Running, status: "False", reason: Error}
|
||||
|
||||
tests:
|
||||
- it: renders when the golden is gone but this pool already cloned it
|
||||
@@ -217,3 +241,13 @@ tests:
|
||||
asserts:
|
||||
- failedTemplate:
|
||||
errorPattern: 'golden DataVolume "talos-worker-sibling-v1\.0\.0" in cozy-public failed to import'
|
||||
|
||||
- it: renders when the golden importer keeps failing but this pool already cloned it
|
||||
set:
|
||||
osImage:
|
||||
builtin: {schematicID: stucklater, version: v1.0.0}
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.dataVolumeTemplates[0].spec.source.pvc.name
|
||||
value: talos-worker-stucklater-v1.0.0
|
||||
documentSelector: {path: kind, value: KubevirtMachineTemplate}
|
||||
@@ -101,10 +101,9 @@ kubernetesProvider:
|
||||
status: {phase: Succeeded}
|
||||
|
||||
tests:
|
||||
# Guaranteed NOT to be a clone, and the case an unresolved empty class waved
|
||||
# through: the pool is on an explicit class, the golden is on the cluster
|
||||
# default, and the two differ. CDI would fall back to the host-assisted copy
|
||||
# over the pod network — silently, which is the transfer this feature removes.
|
||||
# A cross-class clone, and the case an unresolved empty class waved through:
|
||||
# the pool is on an explicit class, the golden is on the cluster default, and
|
||||
# the two differ.
|
||||
- it: fails when the golden takes a cluster default class the pool does not use
|
||||
set:
|
||||
storageClass: local
|
||||
@@ -145,7 +144,7 @@ tests:
|
||||
|
||||
# Reading the spec alone would call this golden "on the cluster default", which
|
||||
# is `replicated` here, and clone it for a `replicated` pool. Its PVC is on
|
||||
# `local`, so that clone would silently be a host-assisted copy.
|
||||
# `local`, so that clone would cross classes.
|
||||
- it: takes a blank-spec golden's class from its PVC, not from today's default
|
||||
set:
|
||||
storageClass: replicated
|
||||
|
||||
@@ -189,7 +189,7 @@ tests:
|
||||
documentSelector: {path: kind, value: KubevirtMachineTemplate}
|
||||
# And the disk binds on the class the guard just compared. Leaving the field
|
||||
# out here passed the guard and then bound the next disk on `fast-nvme`, one
|
||||
# class away from the golden, which is the host-assisted copy the guard is
|
||||
# class away from the golden, which is the cross-class clone the guard is
|
||||
# there to refuse.
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.dataVolumeTemplates[0].spec.storage.storageClassName
|
||||
@@ -243,7 +243,7 @@ tests:
|
||||
version: v1.13.6
|
||||
asserts:
|
||||
- failedTemplate:
|
||||
errorPattern: 'An empty storageClass was resolved to the cluster default StorageClass "fast-nvme"'
|
||||
errorPattern: 'An empty storageClass was resolved to "fast-nvme", the class CDI binds a disk that names none on\.'
|
||||
|
||||
# With both sides explicit nothing was resolved from the default, and the
|
||||
# sentence is absent rather than reporting an empty lookup. The two sentences
|
||||
@@ -257,7 +257,7 @@ tests:
|
||||
version: v1.13.6
|
||||
asserts:
|
||||
- failedTemplate:
|
||||
errorPattern: 'host-assisted copy over the pod network\. Neither side can be moved in place'
|
||||
errorPattern: 'to the CSI driver\. Neither side can be moved in place'
|
||||
|
||||
# The size floor reads the PVC when the DataVolume spec omits a size, so a
|
||||
# golden it could size no longer slips past unmeasured.
|
||||
|
||||
@@ -119,6 +119,54 @@ kubernetesProvider:
|
||||
storageClassName: replicated
|
||||
status:
|
||||
phase: Failed
|
||||
# Goldens whose importer has restarted. CDI leaves an unreachable URL in
|
||||
# progress and restarts the importer instead of reaching Failed, so restarts
|
||||
# plus an importer that is not running now is what marks one as failing.
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata: {name: talos-worker-stuck-v1.0.0, namespace: cozy-public}
|
||||
spec: &goldenSpec
|
||||
storage:
|
||||
resources: {requests: {storage: 6Gi}}
|
||||
storageClassName: replicated
|
||||
status:
|
||||
phase: ImportInProgress
|
||||
# A string: helm-unittest's lookup cannot deep-copy an int, and | int takes either.
|
||||
restartCount: "3"
|
||||
conditions:
|
||||
- {type: Running, status: "False", reason: Error}
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata: {name: talos-worker-recovered-v1.0.0, namespace: cozy-public}
|
||||
spec: *goldenSpec
|
||||
status:
|
||||
phase: ImportInProgress
|
||||
restartCount: "5"
|
||||
conditions:
|
||||
- {type: Running, status: "True", reason: Pod is running}
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata: {name: talos-worker-flaky-v1.0.0, namespace: cozy-public}
|
||||
spec: *goldenSpec
|
||||
status:
|
||||
phase: ImportInProgress
|
||||
restartCount: "2"
|
||||
conditions:
|
||||
- {type: Running, status: "False", reason: Error}
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata: {name: talos-worker-done-v1.0.0, namespace: cozy-public}
|
||||
spec: *goldenSpec
|
||||
status:
|
||||
phase: Succeeded
|
||||
restartCount: "4"
|
||||
conditions:
|
||||
- {type: Running, status: "False", reason: Completed}
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata: {name: talos-worker-norunning-v1.0.0, namespace: cozy-public}
|
||||
spec: *goldenSpec
|
||||
status: {phase: ImportInProgress, restartCount: "3"}
|
||||
# A populated golden that declares no storage request at all. The size guard
|
||||
# reads spec.storage.resources.requests.storage, so its presence gate is the
|
||||
# branch that decides between comparing and skipping; without a fixture the
|
||||
@@ -271,8 +319,56 @@ tests:
|
||||
- failedTemplate:
|
||||
errorPattern: 'golden DataVolume "talos-worker-failed-v1\.0\.0" in cozy-public failed to import \(phase Failed\)'
|
||||
|
||||
# Cross-StorageClass clone: CDI would silently fall back to a host-assisted copy
|
||||
# over the pod network, the exact transfer osImage.builtin exists to avoid.
|
||||
- it: osImage.builtin fails the render when the golden importer keeps failing
|
||||
set:
|
||||
osImage:
|
||||
builtin: {schematicID: stuck, version: v1.0.0}
|
||||
asserts:
|
||||
- failedTemplate:
|
||||
errorPattern: 'golden DataVolume "talos-worker-stuck-v1\.0\.0" in cozy-public keeps failing to import \(3 importer restarts\).*lifts by itself.*a recreated importer pod counts as not running until its container starts'
|
||||
|
||||
- it: osImage.builtin clones a golden whose importer restarted but is running again
|
||||
set:
|
||||
osImage:
|
||||
builtin: {schematicID: recovered, version: v1.0.0}
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.dataVolumeTemplates[0].spec.source.pvc.name
|
||||
value: talos-worker-recovered-v1.0.0
|
||||
documentSelector: {path: kind, value: KubevirtMachineTemplate}
|
||||
|
||||
- it: osImage.builtin clones a golden whose importer failed fewer than three times
|
||||
set:
|
||||
osImage:
|
||||
builtin: {schematicID: flaky, version: v1.0.0}
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.dataVolumeTemplates[0].spec.source.pvc.name
|
||||
value: talos-worker-flaky-v1.0.0
|
||||
documentSelector: {path: kind, value: KubevirtMachineTemplate}
|
||||
|
||||
- it: osImage.builtin clones a golden that Succeeded after its importer restarted
|
||||
set:
|
||||
osImage:
|
||||
builtin: {schematicID: done, version: v1.0.0}
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.dataVolumeTemplates[0].spec.source.pvc.name
|
||||
value: talos-worker-done-v1.0.0
|
||||
documentSelector: {path: kind, value: KubevirtMachineTemplate}
|
||||
|
||||
- it: osImage.builtin clones a golden whose importer restarted and reports no Running condition
|
||||
set:
|
||||
osImage:
|
||||
builtin: {schematicID: norunning, version: v1.0.0}
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.dataVolumeTemplates[0].spec.source.pvc.name
|
||||
value: talos-worker-norunning-v1.0.0
|
||||
documentSelector: {path: kind, value: KubevirtMachineTemplate}
|
||||
|
||||
# Cross-StorageClass clone: a host-assisted copy over the pod network, or a
|
||||
# clone left to the CSI driver to support, so the pool is refused.
|
||||
- it: osImage.builtin fails when the pool storageClass differs from the golden's
|
||||
set:
|
||||
storageClass: local
|
||||
|
||||
@@ -114,3 +114,22 @@ tests:
|
||||
asserts:
|
||||
- failedTemplate:
|
||||
errorPattern: 'requested golden image "talos-worker-elsewhere-v1\.0\.0" not found'
|
||||
|
||||
# CDI reads only the GA annotation when it picks a class itself, so a newer
|
||||
# class carrying only the beta one does not displace a GA default.
|
||||
- it: prefers a GA default over a newer class with only the beta annotation
|
||||
kubernetesProvider:
|
||||
objects:
|
||||
- kind: StorageClass
|
||||
apiVersion: storage.k8s.io/v1
|
||||
metadata:
|
||||
name: beta-only
|
||||
creationTimestamp: "2026-09-01T00:00:00Z"
|
||||
annotations:
|
||||
storageclass.beta.kubernetes.io/is-default-class: "true"
|
||||
asserts:
|
||||
- notFailedTemplate: {}
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.dataVolumeTemplates[0].spec.storage.storageClassName
|
||||
value: new
|
||||
documentSelector: {path: kind, value: KubevirtMachineTemplate}
|
||||
@@ -0,0 +1,183 @@
|
||||
suite: osImage.builtin default-StorageClass selection with a virtualization default
|
||||
|
||||
# CDI resolves a DataVolume with no class and contentType kubevirt to the class
|
||||
# annotated storageclass.kubevirt.io/is-default-virt-class before it looks at the
|
||||
# Kubernetes default, so a cluster that carries both binds the golden and the
|
||||
# worker disks on the virtualization default. The clone guard has to resolve an
|
||||
# empty class the same way, or it compares the golden against a class it is not
|
||||
# on.
|
||||
|
||||
templates:
|
||||
- templates/nodegroup.yaml
|
||||
|
||||
release:
|
||||
name: kubernetes-nodes-myk8s-md0
|
||||
namespace: tenant-test
|
||||
|
||||
set:
|
||||
cluster: myk8s
|
||||
_cluster:
|
||||
cluster-domain: cozy.local
|
||||
version: "v1.35"
|
||||
minReplicas: 1
|
||||
maxReplicas: 3
|
||||
instanceType: u1.medium
|
||||
diskSize: 20Gi
|
||||
storageClass: ""
|
||||
roles: [ingress-nginx]
|
||||
osImage:
|
||||
builtin: {}
|
||||
|
||||
kubernetesProvider:
|
||||
scheme:
|
||||
"instancetype.kubevirt.io/v1beta1/VirtualMachineClusterInstancetype":
|
||||
gvr: {group: instancetype.kubevirt.io, version: v1beta1, resource: virtualmachineclusterinstancetypes}
|
||||
namespaced: false
|
||||
"cluster.x-k8s.io/v1beta1/MachineSet":
|
||||
gvr: {group: cluster.x-k8s.io, version: v1beta1, resource: machinesets}
|
||||
namespaced: true
|
||||
"infrastructure.cluster.x-k8s.io/v1alpha1/KubevirtMachineTemplate":
|
||||
gvr: {group: infrastructure.cluster.x-k8s.io, version: v1alpha1, resource: kubevirtmachinetemplates}
|
||||
namespaced: true
|
||||
"cdi.kubevirt.io/v1beta1/DataVolume":
|
||||
gvr: {group: cdi.kubevirt.io, version: v1beta1, resource: datavolumes}
|
||||
namespaced: true
|
||||
"storage.k8s.io/v1/StorageClass":
|
||||
gvr: {group: storage.k8s.io, version: v1, resource: storageclasses}
|
||||
namespaced: false
|
||||
"v1/PersistentVolumeClaim":
|
||||
gvr: {group: "", version: v1, resource: persistentvolumeclaims}
|
||||
namespaced: true
|
||||
objects:
|
||||
- kind: VirtualMachineClusterInstancetype
|
||||
apiVersion: instancetype.kubevirt.io/v1beta1
|
||||
metadata: {name: u1.medium}
|
||||
- kind: MachineSet
|
||||
apiVersion: cluster.x-k8s.io/v1beta1
|
||||
metadata: {name: ignored-ms, namespace: tenant-test}
|
||||
spec: {template: {spec: {infrastructureRef: {kind: Other, name: ignored}}}}
|
||||
- kind: KubevirtMachineTemplate
|
||||
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
|
||||
metadata: {name: ignored-kmt, namespace: tenant-test}
|
||||
# The Kubernetes default, created after the virtualization default so that
|
||||
# creation order alone would pick it.
|
||||
- kind: StorageClass
|
||||
apiVersion: storage.k8s.io/v1
|
||||
metadata:
|
||||
name: kube-default
|
||||
creationTimestamp: "2026-06-01T00:00:00Z"
|
||||
annotations:
|
||||
storageclass.kubernetes.io/is-default-class: "true"
|
||||
- kind: StorageClass
|
||||
apiVersion: storage.k8s.io/v1
|
||||
metadata:
|
||||
name: virt-default
|
||||
creationTimestamp: "2026-01-01T00:00:00Z"
|
||||
annotations:
|
||||
storageclass.kubevirt.io/is-default-virt-class: "true"
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata:
|
||||
name: talos-worker-ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515-v1.13.6
|
||||
namespace: cozy-public
|
||||
spec:
|
||||
storage:
|
||||
resources: {requests: {storage: 6Gi}}
|
||||
storageClassName: virt-default
|
||||
status: {phase: Succeeded}
|
||||
|
||||
tests:
|
||||
- it: resolves an empty pool storageClass to the virtualization default
|
||||
asserts:
|
||||
- notFailedTemplate: {}
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.dataVolumeTemplates[0].spec.storage.storageClassName
|
||||
value: virt-default
|
||||
documentSelector: {path: kind, value: KubevirtMachineTemplate}
|
||||
|
||||
- it: takes the most recently created of two virtualization defaults
|
||||
kubernetesProvider:
|
||||
objects:
|
||||
- kind: StorageClass
|
||||
apiVersion: storage.k8s.io/v1
|
||||
metadata:
|
||||
name: virt-newer
|
||||
creationTimestamp: "2026-03-01T00:00:00Z"
|
||||
annotations:
|
||||
storageclass.kubevirt.io/is-default-virt-class: "true"
|
||||
asserts:
|
||||
- failedTemplate:
|
||||
errorPattern: 'storageClass "virt-newer" differs from golden image .* storageClass "virt-default".* An empty storageClass was resolved to "virt-newer"'
|
||||
|
||||
- it: ignores a virtualization default annotation that is not "true"
|
||||
kubernetesProvider:
|
||||
objects:
|
||||
- kind: StorageClass
|
||||
apiVersion: storage.k8s.io/v1
|
||||
metadata:
|
||||
name: virt-off
|
||||
creationTimestamp: "2026-05-01T00:00:00Z"
|
||||
annotations:
|
||||
storageclass.kubevirt.io/is-default-virt-class: "false"
|
||||
asserts:
|
||||
- notFailedTemplate: {}
|
||||
|
||||
# Before the virtualization default was read, a pool that has not cloned a
|
||||
# disk yet resolved to the Kubernetes default here and rendered.
|
||||
- it: refuses a pool whose golden is on the Kubernetes default instead
|
||||
set:
|
||||
osImage:
|
||||
builtin: {schematicID: kube, version: v1.0.0}
|
||||
kubernetesProvider:
|
||||
objects:
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata:
|
||||
name: talos-worker-kube-v1.0.0
|
||||
namespace: cozy-public
|
||||
spec:
|
||||
storage:
|
||||
resources: {requests: {storage: 6Gi}}
|
||||
storageClassName: kube-default
|
||||
status: {phase: Succeeded}
|
||||
asserts:
|
||||
- failedTemplate:
|
||||
errorPattern: 'storageClass "virt-default" differs from golden image .* storageClass "kube-default".* An empty storageClass was resolved to "virt-default"'
|
||||
|
||||
# A pool that already cloned reads its class back from its own disk's claim
|
||||
# and never reaches the default lookup.
|
||||
- it: keeps rendering a pool that already cloned from a golden on the Kubernetes default
|
||||
set:
|
||||
osImage:
|
||||
builtin: {schematicID: kube, version: v1.0.0}
|
||||
kubernetesProvider:
|
||||
objects:
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata:
|
||||
name: talos-worker-kube-v1.0.0
|
||||
namespace: cozy-public
|
||||
spec:
|
||||
storage:
|
||||
resources: {requests: {storage: 6Gi}}
|
||||
storageClassName: kube-default
|
||||
status: {phase: Succeeded}
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata:
|
||||
name: disk-system-kubernetes-myk8s-md0-abcde
|
||||
namespace: tenant-test
|
||||
annotations:
|
||||
kubernetes-worker-image.cozystack.io/golden: talos-worker-kube-v1.0.0
|
||||
kubernetes-worker-image.cozystack.io/pool: kubernetes-myk8s-md0
|
||||
status: {phase: Succeeded}
|
||||
- kind: PersistentVolumeClaim
|
||||
apiVersion: v1
|
||||
metadata: {name: disk-system-kubernetes-myk8s-md0-abcde, namespace: tenant-test}
|
||||
spec: {storageClassName: kube-default}
|
||||
asserts:
|
||||
- notFailedTemplate: {}
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.dataVolumeTemplates[0].spec.storage.storageClassName
|
||||
value: kube-default
|
||||
documentSelector: {path: kind, value: KubevirtMachineTemplate}
|
||||
@@ -14,7 +14,7 @@
|
||||
]
|
||||
},
|
||||
"storageClass": {
|
||||
"description": "StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`.",
|
||||
"description": "StorageClass for the worker node system disk. When empty, CDI picks the class: the one annotated `storageclass.kubevirt.io/is-default-virt-class`, else the cluster default. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: the chart refuses a pool whose class differs from its golden's, because across classes CDI either falls back to a host-assisted copy over the pod network (different CSI drivers or volume modes) or leaves a clone into the other class to the CSI driver. Leaving it empty makes the same comparison against the class CDI would pick. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`.",
|
||||
"type": "string",
|
||||
"default": "replicated",
|
||||
"x-kubernetes-validations": [
|
||||
|
||||
@@ -22,7 +22,7 @@ cluster: ""
|
||||
## @section Common Parameters
|
||||
##
|
||||
|
||||
## @param {string} storageClass - StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`.
|
||||
## @param {string} storageClass - StorageClass for the worker node system disk. When empty, CDI picks the class: the one annotated `storageclass.kubevirt.io/is-default-virt-class`, else the cluster default. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: the chart refuses a pool whose class differs from its golden's, because across classes CDI either falls back to a host-assisted copy over the pod network (different CSI drivers or volume modes) or leaves a clone into the other class to the CSI driver. Leaving it empty makes the same comparison against the class CDI would pick. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`.
|
||||
## @x-cozystack-options {source: storageclass}
|
||||
## @immutable
|
||||
storageClass: replicated
|
||||
|
||||
@@ -12,8 +12,17 @@
|
||||
{{- if hasKey .Values.source "image" }}
|
||||
{{- $desiredSource = dict "pvc" (dict "name" (printf "vm-default-images-%s" (required "A valid .Values.source.image.name entry required!" .Values.source.image.name)) "namespace" "cozy-public") }}
|
||||
{{- /* Refuse a golden that is missing or whose import Failed: CDI never
|
||||
populates it, so the clone would wait forever. A golden still importing
|
||||
is left alone because CDI holds the clone until the source is populated.
|
||||
populates it, so the clone would wait forever. Failed is reached only on
|
||||
a fatal error such as a checksum mismatch; an unreachable URL instead
|
||||
restarts the importer with back-off while the phase stays in progress,
|
||||
so an importer that restarted at least three times counts as failing
|
||||
too, running or not: CDI reports every attempt as running, including
|
||||
one that is about to fail again, and this check runs only until the
|
||||
disk exists, so a disk created during an attempt would pass and wait
|
||||
unnoticed. A healthy import with three early restarts is refused until
|
||||
it succeeds, and the release retries the render. Any other golden still
|
||||
importing is left alone because CDI holds the clone until the source is
|
||||
populated.
|
||||
Missing means neither the DataVolume nor the PVC CDI clones exists, as
|
||||
the DataVolume can precede its PVC, and counts only when the release
|
||||
namespace is visible: helm template has no cluster and every lookup
|
||||
@@ -24,8 +33,12 @@
|
||||
{{- if not (or $existingDV (lookup "v1" "PersistentVolumeClaim" .Release.Namespace .Release.Name)) }}
|
||||
{{- $golden := lookup "cdi.kubevirt.io/v1beta1" "DataVolume" "cozy-public" $desiredSource.pvc.name }}
|
||||
{{- $goldenPVC := lookup "v1" "PersistentVolumeClaim" "cozy-public" $desiredSource.pvc.name }}
|
||||
{{- if eq (dig "status" "phase" "" $golden | toString) "Failed" }}
|
||||
{{- $goldenPhase := dig "status" "phase" "" $golden | toString }}
|
||||
{{- $goldenRestarts := dig "status" "restartCount" 0 $golden | int }}
|
||||
{{- if eq $goldenPhase "Failed" }}
|
||||
{{- fail (printf "source.image %q: golden DataVolume %q in cozy-public failed to import (phase Failed), so a clone of it would never complete. Ask the platform administrator to check the importer pod in cozy-public, then delete that DataVolume to retry the import." .Values.source.image.name $desiredSource.pvc.name) }}
|
||||
{{- else if and (ne $goldenPhase "Succeeded") (ge $goldenRestarts 3) }}
|
||||
{{- fail (printf "source.image %q: golden DataVolume %q in cozy-public has not finished importing after %d importer restarts, so a clone of it would not complete while the import keeps failing. The refusal lifts by itself once the import succeeds. The importer is retried with a growing back-off, so after the source is fixed the refusal lasts for the current back-off delay plus the full image download. Ask the platform administrator to check the importer pod logs in cozy-public." .Values.source.image.name $desiredSource.pvc.name $goldenRestarts) }}
|
||||
{{- else if and (not $golden) (not $goldenPVC) (lookup "v1" "Namespace" "" .Release.Namespace) }}
|
||||
{{- fail (printf "source.image %q: golden image %q not found in cozy-public, so a clone of it would never start. Check the image name, or ask the platform administrator to add cozystack.vm-default-images to bundles.enabledPackages." .Values.source.image.name $desiredSource.pvc.name) }}
|
||||
{{- end }}
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
suite: vm-disk refuses to clone a golden image that is missing or failed
|
||||
# A golden that is missing or whose import Failed is never populated, and CDI
|
||||
# holds a clone of it forever, so a new disk on it would sit pending with
|
||||
# nothing pointing at the image. A golden still importing is safe to clone
|
||||
# because CDI waits for it, and helm template has no cluster to look it up in.
|
||||
suite: vm-disk refuses to clone a golden image that is missing, failed or stuck
|
||||
# A golden that is missing, whose import Failed, or whose importer keeps
|
||||
# crashing is not populated, and CDI holds a clone of it until it is, so a new
|
||||
# disk on it would sit pending with nothing pointing at the image. A golden
|
||||
# still importing is safe to clone because CDI waits for it, and helm template
|
||||
# has no cluster to look it up in.
|
||||
release:
|
||||
name: test-vm-disk
|
||||
namespace: tenant-test
|
||||
@@ -185,3 +186,139 @@ tests:
|
||||
- equal:
|
||||
path: spec.source.pvc
|
||||
value: {name: vm-default-images-ubuntu, namespace: cozy-public}
|
||||
|
||||
- it: refuses a new disk on a golden whose importer keeps failing
|
||||
set:
|
||||
source: {image: {name: ubuntu}}
|
||||
kubernetesProvider:
|
||||
scheme: *scheme
|
||||
objects:
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata: {name: vm-default-images-ubuntu, namespace: cozy-public}
|
||||
status: &stuck
|
||||
phase: ImportInProgress
|
||||
# A string: helm-unittest's lookup cannot deep-copy an int, and | int takes either.
|
||||
restartCount: "3"
|
||||
conditions:
|
||||
- {type: Bound, status: "True"}
|
||||
- {type: Running, status: "False", reason: Error}
|
||||
asserts:
|
||||
- failedTemplate:
|
||||
errorPattern: 'golden DataVolume "vm-default-images-ubuntu" in cozy-public has not finished importing after 3 importer restarts.*lifts by itself once the import succeeds'
|
||||
|
||||
# CDI reports Running=True for every importer attempt, including one that is
|
||||
# about to fail again, and this check runs only until the disk exists.
|
||||
- it: refuses a new disk on a golden that restarted and is running again
|
||||
set:
|
||||
source: {image: {name: ubuntu}}
|
||||
kubernetesProvider:
|
||||
scheme: *scheme
|
||||
objects:
|
||||
- kind: Namespace
|
||||
apiVersion: v1
|
||||
metadata: {name: tenant-test}
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata: {name: vm-default-images-ubuntu, namespace: cozy-public}
|
||||
status:
|
||||
phase: ImportInProgress
|
||||
restartCount: "5"
|
||||
conditions:
|
||||
- {type: Running, status: "True", reason: Pod is running}
|
||||
- kind: PersistentVolumeClaim
|
||||
apiVersion: v1
|
||||
metadata: {name: vm-default-images-ubuntu, namespace: cozy-public}
|
||||
asserts:
|
||||
- failedTemplate:
|
||||
errorPattern: 'has not finished importing after 5 importer restarts'
|
||||
|
||||
- it: clones a golden whose importer failed fewer than three times
|
||||
set:
|
||||
source: {image: {name: ubuntu}}
|
||||
kubernetesProvider:
|
||||
scheme: *scheme
|
||||
objects:
|
||||
- kind: Namespace
|
||||
apiVersion: v1
|
||||
metadata: {name: tenant-test}
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata: {name: vm-default-images-ubuntu, namespace: cozy-public}
|
||||
status:
|
||||
phase: ImportInProgress
|
||||
restartCount: "2"
|
||||
conditions:
|
||||
- {type: Running, status: "False", reason: Error}
|
||||
- kind: PersistentVolumeClaim
|
||||
apiVersion: v1
|
||||
metadata: {name: vm-default-images-ubuntu, namespace: cozy-public}
|
||||
asserts:
|
||||
- notFailedTemplate: {}
|
||||
|
||||
- it: clones a golden that Succeeded after its importer restarted
|
||||
set:
|
||||
source: {image: {name: ubuntu}}
|
||||
kubernetesProvider:
|
||||
scheme: *scheme
|
||||
objects:
|
||||
- kind: Namespace
|
||||
apiVersion: v1
|
||||
metadata: {name: tenant-test}
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata: {name: vm-default-images-ubuntu, namespace: cozy-public}
|
||||
status:
|
||||
phase: Succeeded
|
||||
restartCount: "4"
|
||||
conditions:
|
||||
- {type: Running, status: "False", reason: Completed}
|
||||
- kind: PersistentVolumeClaim
|
||||
apiVersion: v1
|
||||
metadata: {name: vm-default-images-ubuntu, namespace: cozy-public}
|
||||
asserts:
|
||||
- notFailedTemplate: {}
|
||||
|
||||
- it: leaves an existing disk alone when its golden is stuck
|
||||
set:
|
||||
source: {image: {name: ubuntu}}
|
||||
kubernetesProvider:
|
||||
scheme: *scheme
|
||||
objects:
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata: {name: vm-default-images-ubuntu, namespace: cozy-public}
|
||||
status: *stuck
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata:
|
||||
name: test-vm-disk
|
||||
namespace: tenant-test
|
||||
annotations:
|
||||
vm-disk.cozystack.io/source: '{"image":{"name":"ubuntu"}}'
|
||||
vm-disk.cozystack.io/storage-class: replicated
|
||||
spec:
|
||||
source: {pvc: {name: vm-default-images-ubuntu, namespace: cozy-public}}
|
||||
storage: {resources: {requests: {storage: 5Gi}}, storageClassName: replicated}
|
||||
asserts:
|
||||
- notFailedTemplate: {}
|
||||
|
||||
- it: refuses a new disk on a golden whose importer restarted and reports no Running condition
|
||||
set:
|
||||
source: {image: {name: ubuntu}}
|
||||
kubernetesProvider:
|
||||
scheme: *scheme
|
||||
objects:
|
||||
- kind: Namespace
|
||||
apiVersion: v1
|
||||
metadata: {name: tenant-test}
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata: {name: vm-default-images-ubuntu, namespace: cozy-public}
|
||||
status: {phase: ImportInProgress, restartCount: "3"}
|
||||
- kind: PersistentVolumeClaim
|
||||
apiVersion: v1
|
||||
metadata: {name: vm-default-images-ubuntu, namespace: cozy-public}
|
||||
asserts:
|
||||
- failedTemplate:
|
||||
errorPattern: 'has not finished importing after 3 importer restarts'
|
||||
@@ -1115,6 +1115,48 @@ tests:
|
||||
asserts:
|
||||
- notFailedTemplate: {}
|
||||
|
||||
- it: leaves a disk alone whose PVC exists without its DataVolume when the golden is stuck
|
||||
set:
|
||||
source: {image: {name: ubuntu}}
|
||||
storageClass: local
|
||||
kubernetesProvider:
|
||||
scheme: *scheme
|
||||
objects:
|
||||
- *namespace
|
||||
- *diskPVC
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata: {name: vm-default-images-ubuntu, namespace: cozy-public}
|
||||
status: &stuck
|
||||
phase: ImportInProgress
|
||||
restartCount: "3"
|
||||
conditions:
|
||||
- {type: Running, status: "False", reason: Error}
|
||||
asserts:
|
||||
- notFailedTemplate: {}
|
||||
|
||||
# Fixing the class alone would still leave the clone waiting on the import.
|
||||
- it: names the stuck import, not the class, for a disk that trips both
|
||||
set:
|
||||
source: {image: {name: ubuntu}}
|
||||
storageClass: local
|
||||
kubernetesProvider:
|
||||
scheme: *scheme
|
||||
objects:
|
||||
- *namespace
|
||||
- *goldenPVC
|
||||
- *cdiCSIClone
|
||||
- *replicatedClass
|
||||
- *localClass
|
||||
- *localProfile
|
||||
- kind: DataVolume
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
metadata: {name: vm-default-images-ubuntu, namespace: cozy-public}
|
||||
status: *stuck
|
||||
asserts:
|
||||
- failedTemplate:
|
||||
errorPattern: 'golden DataVolume "vm-default-images-ubuntu" in cozy-public has not finished importing after 3 importer restarts'
|
||||
|
||||
# A key under another prefix belongs to someone else and is not read.
|
||||
- it: clones into a LINSTOR class that allows remote access next to a foreign key that denies it
|
||||
set:
|
||||
|
||||
@@ -765,10 +765,10 @@ backupStrategyController: {}
|
||||
# cozystack.kubernetes-worker-image to bundles.enabledPackages -- the package is
|
||||
# opt-in and this key alone does not enable it. The two knobs that decide
|
||||
# whether the feature is usable at all are imageFactoryURL (an air-gapped or
|
||||
# rate-limited cluster must repoint the single import) and storageClass (CDI
|
||||
# cannot clone across StorageClasses, so a cluster whose worker pools are not on
|
||||
# `replicated` must move the golden onto their class). Empty (the default)
|
||||
# forwards nothing and keeps the child chart defaults.
|
||||
# rate-limited cluster must repoint the single import) and storageClass (a
|
||||
# worker pool refuses a golden on another StorageClass, so a cluster whose
|
||||
# worker pools are not on `replicated` must move the golden onto their class).
|
||||
# Empty (the default) forwards nothing and keeps the child chart defaults.
|
||||
kubernetesWorkerImage: {}
|
||||
# Pod scheduling configuration
|
||||
scheduling:
|
||||
|
||||
@@ -48,7 +48,7 @@ rules:
|
||||
- apiGroups: ["kubevirt.io"]
|
||||
resources: ["kubevirts"]
|
||||
verbs: ["get", "watch", "list"]
|
||||
# The image picker drops golden images whose import Failed.
|
||||
# The image picker drops golden images whose import failed or keeps failing.
|
||||
- apiGroups: ["cdi.kubevirt.io"]
|
||||
resources: ["datavolumes"]
|
||||
verbs: ["get", "watch", "list"]
|
||||
|
||||
@@ -21,7 +21,7 @@ spec:
|
||||
singular: kubernetesnodes
|
||||
plural: kubernetesnodeses
|
||||
openAPISchema: |-
|
||||
{"title":"Chart Values","type":"object","properties":{"cluster":{"description":"Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-<cluster>`; the KubernetesNodes CR must be named `<cluster>-<pool>`. The pool's objects are named `kubernetes-<cluster>-<pool>`; when a MachineDeployment of that name already exists and belongs to another Helm release, the render fails instead of taking it over.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"cluster is immutable"}]},"storageClass":{"description":"StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"minReplicas":{"description":"Minimum number of replicas in this pool. Used as the cluster-autoscaler floor.","type":"integer","default":0},"maxReplicas":{"description":"Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.","type":"integer","default":10},"instanceType":{"description":"KubeVirt instance type that sizes each worker VM when `resources` is not set. Ignored when `resources.cpu` and `resources.memory` are both set, and therefore on `substrate: proxmox`, which requires them.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. Has no effect on `substrate: proxmox`, where the worker disk is a clone of the VM template's disk and keeps its size.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"roles":{"description":"List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/<r>`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.","type":"array","default":[],"items":{"type":"string"}},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `iLine truncated
|
||||
{"title":"Chart Values","type":"object","properties":{"cluster":{"description":"Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-<cluster>`; the KubernetesNodes CR must be named `<cluster>-<pool>`. The pool's objects are named `kubernetes-<cluster>-<pool>`; when a MachineDeployment of that name already exists and belongs to another Helm release, the render fails instead of taking it over.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"cluster is immutable"}]},"storageClass":{"description":"StorageClass for the worker node system disk. When empty, CDI picks the class: the one annotated `storageclass.kubevirt.io/is-default-virt-class`, else the cluster default. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: the chart refuses a pool whose class differs from its golden's, because across classes CDI either falls back to a host-assisted copy over the pod network (different CSI drivers or volume modes) or leaves a clone into the other class to the CSI driver. Leaving it empty makes the same comparison against the class CDI would pick. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"minReplicas":{"description":"Minimum number of replicas in this pool. Used as the cluster-autoscaler floor.","type":"integer","default":0},"maxReplicas":{"description":"Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.","type":"integer","default":10},"instanceType":{"description":"KubeVirt instance type that sizes each worker VM when `resources` is not set. Ignored when `resources.cpu` and `resources.memory` are both set, and therefore on `substrate: proxmox`, which requires them.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. Has no effect on `substrate: proxmox`, where the worker disk is a clone of the VM template's disk and keeps its size.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"roles":{"description":"List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/<r>`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.","type":"array","default":[],"items":{"type":"string"}},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When bLine truncated
|
||||
release:
|
||||
labels:
|
||||
sharding.fluxcd.io/key: tenants
|
||||
|
||||
@@ -30,10 +30,12 @@
|
||||
## `talos-worker-<schematicID>-<version>` from cozy-public. The tenant render fails
|
||||
## rather than degrade silently when the referenced (schematicID, version) has no
|
||||
## matching entry in this catalog, when its golden terminally failed to import
|
||||
## (`phase: Failed`), or when the node group's storageClass differs from the
|
||||
## (`phase: Failed`) or keeps failing to (its importer restarted at least 3 times
|
||||
## and is not running), or when the node group's storageClass differs from the
|
||||
## golden's — so keep the catalog in sync with the (schematicID, version) pairs and
|
||||
## StorageClasses that tenant node groups reference. A golden that is merely
|
||||
## mid-import is not an error: CDI holds the worker disk until the source populates.
|
||||
## StorageClasses that tenant node groups reference. A golden still importing, with
|
||||
## its importer running or restarted fewer than 3 times, is not an error: CDI holds
|
||||
## the worker disk until the source populates.
|
||||
## Goldens carry `helm.sh/resource-policy: keep`, so removing an `images[]` entry
|
||||
## stops managing that golden rather than deleting it. Deleting one by hand while a
|
||||
## worker pool still clones it fails that pool's render on its next values or chart
|
||||
@@ -88,7 +90,7 @@
|
||||
## @param {string} imageFactoryURL - Default base URL of the Talos Image Factory serving the raw worker OS image (`<url>/image/<schematicID>/<version>/openstack-amd64.raw.xz`). Overridable per image. No trailing slash.
|
||||
imageFactoryURL: "https://factory.talos.dev"
|
||||
|
||||
## @param {string} storageClass - Default StorageClass for the golden image(s). Must match the StorageClass of the worker disks that clone it: CDI cannot CSI-clone across StorageClasses and would silently fall back to a host-assisted copy over the pod network, so the tenant render rejects a mismatch outright. Overridable per image.
|
||||
## @param {string} storageClass - Default StorageClass for the golden image(s). Must match the StorageClass of the worker disks that clone it: across classes CDI either falls back to a host-assisted copy over the pod network or leaves the clone to the CSI driver, so the tenant render rejects a mismatch outright. Overridable per image.
|
||||
storageClass: "replicated"
|
||||
|
||||
## @param {quantity} storage - Default storage size allocated for each golden image. Must be >= the imported raw image virtual size and <= the smallest worker diskSize that clones it. The Talos v1.13.x openstack raw is ~4.15 GiB, so 6Gi leaves headroom. This size is also the unit of the transient cost every clone pays: CDI populates a worker disk larger than the golden by cloning at the golden's size and then growing it, so one worker in flight occupies the golden plus a temporary clone PVC plus its own diskSize, times the replica count of a replicated StorageClass. Keeping this at the raw image's size rather than rounding it up is therefore worth doing -- it is paid once per golden and again per concurrent clone. Overridable per image, but only when the entry is added: a DataVolume spec cannot be patched in place, so raising this on an entry whose golden already exists is rejected on the next upgrade — a Talos version whose raw outgrows it needs a new `images[]` entry, which it gets anyway since the name is keyed on the version.
|
||||
|
||||
@@ -9,7 +9,7 @@
|
||||
## images that did import stay usable meanwhile.
|
||||
##
|
||||
|
||||
## @param {string} storageClass - Default StorageClass for all images. If empty, uses the cluster default StorageClass.
|
||||
## @param {string} storageClass - Default StorageClass for all images. If empty, CDI picks the class: the one annotated `storageclass.kubevirt.io/is-default-virt-class`, else the cluster default.
|
||||
## NOTE: The default set of images requires approximately 320Gi of storage (16 images × 20Gi each).
|
||||
## Adjust the image list or per-image storage sizes to match your cluster capacity.
|
||||
storageClass: "replicated"
|
||||
|
||||
@@ -335,9 +335,10 @@ func permittedResourceNames(kv *unstructured.Unstructured) map[string]struct{} {
|
||||
// imageProvider lists the default image PVCs in cozy-public and strips the
|
||||
// vm-default-images- prefix to get the catalog name used by the vm-disk chart.
|
||||
//
|
||||
// A golden whose DataVolume is in phase Failed is left out: CDI never
|
||||
// populates it, so a disk cloned from it never becomes usable. One still
|
||||
// importing stays, because CDI holds a clone until the source is populated.
|
||||
// A golden whose import is failing is left out (see goldenImportFailing):
|
||||
// CDI does not populate it, so a disk cloned from it never becomes usable.
|
||||
// One still importing stays, because CDI holds a clone until the source is
|
||||
// populated.
|
||||
func imageProvider(dyn dynamic.Interface) providerFunc {
|
||||
return func(ctx context.Context, _ string) ([]corev1alpha1.OptionItem, error) {
|
||||
list, err := dyn.Resource(gvrPVCs).Namespace(publicImagesNamespace).List(ctx, listOpts())
|
||||
@@ -353,7 +354,7 @@ func imageProvider(dyn dynamic.Interface) providerFunc {
|
||||
logProviderError("image", err)
|
||||
} else {
|
||||
for i := range dvs.Items {
|
||||
if phase, _, _ := unstructured.NestedString(dvs.Items[i].Object, "status", "phase"); phase == "Failed" {
|
||||
if goldenImportFailing(&dvs.Items[i]) {
|
||||
failed[dvs.Items[i].GetName()] = true
|
||||
}
|
||||
}
|
||||
@@ -371,6 +372,33 @@ func imageProvider(dyn dynamic.Interface) providerFunc {
|
||||
}
|
||||
}
|
||||
|
||||
// goldenImportFailing reports a golden DataVolume whose import is failing.
|
||||
// CDI sets phase Failed only on a fatal error such as a checksum mismatch; an
|
||||
// unreachable URL restarts the importer with back-off while the phase stays
|
||||
// in progress, so an importer that restarted at least three times and is not
|
||||
// running now counts as failing too. The kubernetes-nodes chart applies the
|
||||
// same test before cloning a golden; vm-disk refuses on the restarts alone.
|
||||
func goldenImportFailing(dv *unstructured.Unstructured) bool {
|
||||
phase, _, _ := unstructured.NestedString(dv.Object, "status", "phase")
|
||||
if phase == "Failed" {
|
||||
return true
|
||||
}
|
||||
if phase == "Succeeded" {
|
||||
return false
|
||||
}
|
||||
restarts, _, _ := unstructured.NestedInt64(dv.Object, "status", "restartCount")
|
||||
if restarts < 3 {
|
||||
return false
|
||||
}
|
||||
conditions, _, _ := unstructured.NestedSlice(dv.Object, "status", "conditions")
|
||||
for _, c := range conditions {
|
||||
if cm, ok := c.(map[string]any); ok && cm["type"] == "Running" {
|
||||
return cm["status"] == "False"
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// storagePoolProvider derives selectable pool names from the seaweedfs
|
||||
// HelmRelease(s) in the request namespace (volume.pools and
|
||||
// volume.zones.<zone>.pools map keys). Best-effort: returns nil when no
|
||||
|
||||
@@ -344,6 +344,41 @@ func TestImageProviderSkipsGoldensWhoseImportFailed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestImageProviderSkipsGoldensWhoseImporterKeepsFailing(t *testing.T) {
|
||||
pvcGVK := gvrPVCs.GroupVersion().WithKind("PersistentVolumeClaim")
|
||||
dvGVK := gvrDataVolumes.GroupVersion().WithKind("DataVolume")
|
||||
dv := func(name, phase string, restarts int64, running string) *unstructured.Unstructured {
|
||||
o := newObj(dvGVK, publicImagesNamespace, name, nil)
|
||||
_ = unstructured.SetNestedField(o.Object, phase, "status", "phase")
|
||||
_ = unstructured.SetNestedField(o.Object, restarts, "status", "restartCount")
|
||||
if running != "" {
|
||||
_ = unstructured.SetNestedSlice(o.Object, []any{
|
||||
map[string]any{"type": "Bound", "status": "True"},
|
||||
map[string]any{"type": "Running", "status": running},
|
||||
}, "status", "conditions")
|
||||
}
|
||||
return o
|
||||
}
|
||||
golden := func(name string) *unstructured.Unstructured {
|
||||
return newObj(pvcGVK, publicImagesNamespace, "vm-default-images-"+name, nil)
|
||||
}
|
||||
dyn := dynamicfake.NewSimpleDynamicClientWithCustomListKinds(runtime.NewScheme(), listKinds(),
|
||||
golden("stuck"), golden("recovered"), golden("flaky"), golden("done"), golden("norunning"),
|
||||
dv("vm-default-images-stuck", "ImportInProgress", 3, "False"),
|
||||
dv("vm-default-images-recovered", "ImportInProgress", 5, "True"),
|
||||
dv("vm-default-images-flaky", "ImportInProgress", 2, "False"),
|
||||
dv("vm-default-images-done", "Succeeded", 4, "False"),
|
||||
dv("vm-default-images-norunning", "ImportInProgress", 3, ""),
|
||||
)
|
||||
items, err := DefaultProviders(dyn)["image"](context.Background(), "")
|
||||
if err != nil {
|
||||
t.Fatalf("image provider: %v", err)
|
||||
}
|
||||
if got := values(items); len(got) != 4 || got[0] != "done" || got[1] != "flaky" || got[2] != "norunning" || got[3] != "recovered" {
|
||||
t.Fatalf("image: got %v, want [done flaky norunning recovered] (stuck golden dropped)", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestImageProviderListsGoldensWhenDataVolumesCannotBeRead(t *testing.T) {
|
||||
for name, listErr := range map[string]error{
|
||||
"no CDI": apierrors.NewNotFound(gvrDataVolumes.GroupResource(), ""),
|
||||
|
||||
Reference in new issue
Block a user