Commit Graph
676 Commits
Author SHA1 Message Date
ZverGuy 95dc591198 test(kubernetes): keep application identity out of guest log streams
Guest pods are tenant-controlled, so their lineage labels are not a
trustworthy application identity, and the platform must not promote
them to the stream fields the parent scopes application logs by. The
guest release replaces the monitoring-agents filters and outputs
wholesale, which is what keeps the parent's label lift out of guests;
pin that so a later change to the guest overrides cannot bring it in.

This only covers what the platform ships. A guest admin can still edit
the guest's fluent-bit or write to the parent's vlinsert directly; that
needs a write-path control, which the logs proposal leaves open.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-04 04:18:09 +07:00
myasnikovdaniil 4087218a02 docs: move the repo-only guides onto the website (#5034)
## What this PR does

Seven guides under `docs/` were operator facing and written alongside
the code, and left in the source tree where no user looks. They are on
the website now, so this deletes them here and repoints every reference.

```
docs/operations/backup-classes.md                  -> operations/services/backup-classes
docs/operations/resource-presets.md                -> guides/resource-management
docs/storage-immutability.md                       -> operations/storage-class-immutability
docs/operations/system-memory-limits.md            -> operations/configuration/system-memory-limits
docs/gateway-classes.md                            -> networking/gateway-classes
docs/observability/distributed-tracing.md          -> operations/services/monitoring/distributed-tracing
docs/operations/seaweedfs-capacity-for-backups.md  -> operations/services/object-storage/capacity-for-backups
```

`docs/release.md`, `docs/deprecating-applications.md` and the two
SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`,
a migration and a bats suite read them by path, which is the test from
#3777.

References point at `https://cozystack.io/docs/v1/...`. The site
rewrites that short prefix to the latest version in `layouts/404.html`,
so the links follow releases and pin none. Backup Classes, resource
management and `storageClass` immutability exist in the latest version
already. The other four pages are only in `next/` until 1.7.0 is
promoted, and their links start to resolve then. The generated mongodb
files come from `values.yaml` through `make generate` with
cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`.

Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114)
and kubernetes-nodes (223), go tests for backupcontroller, presets and
the operator, rd-presets-check, the migration-50 bats suite.

### Downstream repositories

- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#763

Walked the rest of the trigger map against the diff. No provider
follow-up: the mongodb `values.schema.json` diff is one `description`
string, with no field added, removed or renamed, no enum, no default,
and no `kind`, `plural` or `release.prefix` change.
`packages/core/platform/values.yaml` changed in comments only and no
key, so the hand written platform-package table on the site is
unaffected.

Please merge after cozystack/website#763, the links in the code only
resolve once those pages deploy.

### Release note

```release-note
docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there.
```
2026-11-03 23:09:46 +05:00
IvanHunters 8d96c46994 fix(monitoring-agents): isolate merged log payload under log_processed
Fluent Bit's kubernetes filter runs with Merge_Log On but without
Merge_Log_Key, so a parsed JSON log line is merged into the record root.
An application log line can thereby forge the service fields that later
pipeline stages rely on for routing and tenant isolation
(kubernetes_namespace_name, kubernetes.labels, tenant, cluster,
log_source). Setting Merge_Log_Key log_processed nests the parsed
payload under a dedicated key so application content can no longer
override top-level service fields.

Applies to both the platform (system/monitoring-agents) and guest
(apps/kubernetes) Fluent Bit configurations.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-10-29 17:08:49 +03:00
Aleksei Sviridkin 44848b9c9d fix(kubernetes): run Proxmox CCM v0.16.1 and pin the Proxmox images by digest (#4917)
## What this PR does

`v1.7.0-rc.1` and `release-1.7` ship the Proxmox
cloud-controller-manager as
`ghcr.io/sergelogvinov/proxmox-cloud-controller-manager:v0.15.0`, a bare
tag. That tag was re-pushed upstream on 2026-09-27: it used to name the
2026-08-06 build (`sha256:fbc553921145…`) and now names
`sha256:d81702998ae1…`, a binary whose gitVersion is `0.15.0` without
the leading `v`. component-base v0.37 parses that string in an `init()`
and panics:

```
panic: version string "0.15.0" doesn't match expected regular expression: "^v(\d+\.\d+\.\d+)"
```

The CCM Deployment crash-loops, no new Node loses
`node.cloudprovider.kubernetes.io/uninitialized`, and workloads on new
workers stay Pending. On a live Proxmox cluster this went unnoticed for
five days: the existing Nodes were already initialized, and the failure
only showed when the next new worker came up. Every fresh install pulls
the broken build, so as it stands v1.7.0 would be the first stable
release with the Proxmox substrate, shipping a CCM that cannot start.

Two commits, both cherry-pick onto `release-1.7` unchanged:

1. **`fix(kubernetes)`**: run v0.16.1, pinned as tag plus digest in
`images/proxmox-cloud-controller-manager.tag`. That is the same shape
the chart already uses for the KubeVirt cloud provider. v0.16 builds
with goreleaser and stamps the version with its `v`. The config the init
container writes still parses: the cluster entry moved to
go-proxmox-pool's `ClusterConfig` with the same `url`, `insecure`,
`token_id`, `token_secret` and `region` keys, and `features.provider:
capmox` still writes `proxmox://<SMBIOS UUID>`, the providerID capmox
puts on the Machine.
2. **`fix(kubernetes, proxmox-csi)`**: pin the CSI driver from the same
registry at its current v0.20.0, with no version change. The
management-side controller goes through
`images/proxmox-csi-controller.tag`. The tenant-side node plugin goes
through `repository` + `tag` in
`packages/system/proxmox-csi/values.yaml`, which also puts it in
`hack/lib/image-refs.sh` and so in front of the release multi-arch gate.
Pinning takes the version out of the vendored chart's hands, so
`hack/proxmox-csi-driver-pin_test.bats` now fails unless the chart's
`appVersion` and both pins name the same release.

All three digests are OCI indexes for `linux/amd64` and `linux/arm64`.
The sig-storage sidecars stay on tags, as they do for the KubeVirt CSI
driver: registry.k8s.io publishes through image promotion, which does
not move a tag.

### Testing

- `helm unittest`: `packages/apps/kubernetes` 31 suites / 346 tests,
`packages/apps/kubernetes-nodes` 37 / 223, all passing.
- bats: the new test, plus `image-pin-consistency`,
`promote-rewrite-tags_test`, `overlay-main-images_test`,
`nightly-mirror_test`, `promote-retag_test`, `stitch-multiarch_test`,
`bats-strict-setup`, `bats-runner-coverage`: 107/107. The new test was
checked to fail on a node tag at another version.
- `git cherry-pick -x` of both commits onto `release-1.7` (315b335cf)
applies cleanly, and the Proxmox unit suites and the new bats test pass
there.
- v0.16.1 compatibility was checked against its source, as described
above. It has also run on a live Proxmox cluster, including the
initialisation of a new worker; the results are in the comment below.

### Screenshots

Not a UI change.

### Downstream repositories

The diff touches no app `values.schema.json` or defaults, adds or
renames no package, and moves nothing under `hack/`.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kubernetes): the Proxmox cloud-controller-manager now runs v0.16.1, and the Proxmox CCM and CSI driver images are pinned by digest. The v0.15.0 tag referenced before was re-pushed upstream to a build that panics on start, which left every new Proxmox worker with the uninitialized taint.
```
2026-10-29 11:20:22 +03:00
Marian Koreniuk 780b068df0 fix(kubernetes, proxmox-csi): pin the Proxmox CSI driver images by digest
The CSI controller in the management cluster was referenced by the bare
tag ghcr.io/sergelogvinov/proxmox-csi-controller:v0.20.0, and the node
plugin in the tenant by the vendored chart's default, its appVersion as
a bare tag. That registry is the one whose proxmox-cloud-controller-
manager:v0.15.0 was re-pushed to a build that panics, so a tag there is
not a statement of what runs.

Both now carry the v0.20.0 digest: the controller through
images/proxmox-csi-controller.tag, like the cloud-controller-manager, and
the node plugin through repository and tag in the proxmox-csi values,
which also makes it visible to hack/lib/image-refs.sh and therefore to
the release multi-arch gate. Both digests are indexes for linux/amd64
and linux/arm64.

Pinning takes the version out of the vendored chart's hands: a chart
bump would bring a new appVersion while both pins kept naming the old
release, and the controller and the node plugin would drift apart from
the templates and from each other without anything noticing.
hack/proxmox-csi-driver-pin_test.bats fails unless the chart's
appVersion and both pins name the same release.

The sig-storage sidecars stay on tags, as they do for the KubeVirt CSI
driver: registry.k8s.io publishes them through image promotion, which
does not overwrite a tag.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
2026-10-28 18:38:35 +01:00
Marian Koreniuk 05a347e6f6 fix(kubernetes): run proxmox-cloud-controller-manager v0.16.1, pinned by digest
The Proxmox CCM was referenced by its bare v0.15.0 tag, and that tag no
longer names the image it named when the template was written. It was
re-pushed on 2026-09-27 (index sha256:d81702998ae1, replacing the
2026-08-06 build sha256:fbc553921145) to a binary whose gitVersion is
"0.15.0" without the leading v. component-base v0.37 parses that string
in the init() of its legacy metrics registry and panics:

  panic: version string "0.15.0" doesn't match expected regular
  expression: "^v(\d+\.\d+\.\d+)"

so the Deployment crash-loops, no Node ever loses the
node.cloudprovider.kubernetes.io/uninitialized taint, and workloads on a
new worker stay Pending. Any node that pulls the tag fresh gets it, which
is every fresh install.

v0.16 builds with goreleaser and stamps gitVersion as v{{.Version}}. The
configuration this template writes is read unchanged: the cluster entry
moved to go-proxmox-pool's ClusterConfig with the same url, insecure,
token_id, token_secret and region keys, and features.provider: capmox
still writes proxmox://<SMBIOS UUID>, the providerID capmox puts on the
Machine.

The reference now lives in images/proxmox-cloud-controller-manager.tag
as tag plus digest, the shape the chart already uses for the KubeVirt
cloud provider, so a moving tag can no longer change what runs.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
2026-10-28 18:38:35 +01:00
Aleksei Sviridkin ae9e7c4c64 test(kubernetes): pin RetryOnFailure on the Proxmox csi release (#4969)
The KubeVirt csi and cilium releases carry a helm-unittest pin on
`strategy.name: RetryOnFailure` for both actions, and the e2e
HelmRelease guard in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` names
those two tests as load-bearing. `helmreleases/csi-proxmox.yaml` sets
the same strategy for the same reason and had no pin.

This adds
`packages/apps/kubernetes/tests/csi_proxmox_install_retry_test.yaml`, a
mirror of `csi_install_retry_test.yaml` rendered on the proxmox
substrate, and updates the guard comment that pointed at the gap. No
template change.

Verified with `helm unittest .` in `packages/apps/kubernetes` (32
suites, 349 tests).

Closes #4731


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
* Expanded automated coverage for Proxmox CSI installation and upgrade
retry behavior, including the retry interval and remediation retry
limits. The checks verify that retries continue according to the
configured settings during installation and upgrades. This update does
not change user-facing functionality.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-28 18:30:06 +03:00
Andrei Kvapil b43196e549 fix(kubernetes): pin the Proxmox cloud-config builders to the bundled image
The Proxmox substrate added two init containers that build the cloud
config from the hypervisor token Secret, and both still took their
image from images.waitForKubeconfig. With that key gone the render
failed on a nil map; with it set, a tenant chose the image that reads
the token. They now run the bundled busybox through cozy-lib.image, like
the wait-for-kubeconfig container next to them.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:17 +05:00
Andrei Kvapil 1636349a63 fix(api): point the removed image overrides at node registry mirrors
The warning for a removed image override told an air-gapped install to
use the platform-wide registry, but nothing in the platform writes
_cluster.images-registry, so there is nothing to set. What serves these
images offline is what serves every other platform image: the registry
mirrors configured on the nodes, as the air-gapped install guide sets
up. The tests that hold the cozy-lib.image routing now say what they
check rather than promising a platform knob.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:16 +05:00
Andrei Kvapil dfff5c2dac test(kubernetes): hold the routing of the busybox and unpin images
The busybox helper shared by four workloads and the pool's pre-delete
Job had no test holding their route through cozy-lib.image, so
reverting either line left both suites green.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:16 +05:00
Andrei Kvapil d99a1b9f2f fix(kubernetes): route the helper images through cozy-lib.image
These images now go through cozy-lib.image like the harbor, kafka and
mariadb hook images, so they follow _cluster.images-registry wherever
that is set rather than any per-application value. An air-gapped
install serves them the way it serves every other platform image,
through the registry mirrors configured on its nodes.

The talos-csr-signer sidecar and the OIDC Jobs had no test holding
their image, which is where an override could come back unnoticed.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:15 +05:00
Andrei Kvapil af20694359 fix(kubernetes)!: drop the helper image overrides
images.waitForKubeconfig, images.kubectl and images.talosCsrSigner
ran whatever image a tenant named inside the management cluster: the
csr signer as a sidecar of the tenant control plane with the Talos CA
key mounted, the other two in pods holding service account tokens for
CAPI, KubeVirt and Cilium objects in the tenant namespace. A tenant
never gets that access through the apps API, and the platform should
not hand it over through an image field.

The images now always come from the bundled tag files. A release that
still carries the keys renders exactly as one without them.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:14 +05:00
Myasnikov Daniil b83cc087cc docs: move the repo-only guides onto the website
Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.

docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-21 17:50:46 +05:00
Marian Koreniuk 98d0402096 test(kubernetes): pin RetryOnFailure on the Proxmox csi release
The KubeVirt csi and cilium releases carry a helm-unittest pin on
strategy.name: RetryOnFailure for both actions, and the e2e HelmRelease
guard names those two tests as load-bearing. helmreleases/csi-proxmox.yaml
sets the same strategy for the same reason and had no pin. Add the mirror
test and update the guard comment that pointed at the gap.

No template change.

Closes #4731

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-10-19 23:08:02 +02:00
Myasnikov Daniil f3486f4e3a docs: move the operator guides out of docs/ to where readers find them
Five guides in docs/ were written next to the feature code that needed
them and left in the source tree, where no operator looks. They are on
the website, so a second copy here only drifts: edits kept landing in
docs/ after the website copy was made.

docs/vm-external-vlan.md is split rather than moved. Its section on
staging the reference CNI plugins is a contract that
hack/cni-plugins-staging-contract.bats asserts against the multus
Dockerfile, the daemonset manifest and the platform values, so that half
is package documentation and goes to packages/system/multus/README.md,
which the suite now reads. The operator half goes to the website.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-14 16:24:16 +05:00
Aleksei Sviridkin 953b965e66 fix(kubernetes): keep NFS publish from hanging on a dead server (#4791)
## What this PR does

When the kubevirt-csi node plugin publishes an NFS volume, it first
mounts the export root into an `nfs-init-*` temp dir to create `/data`
and move old files into it. That temp mount was hard. If the NFS server
stopped answering at that point, the publish call never returned.
kubelet gives up after 2m and retries, each retry blocked the same way,
and goroutines and temp dirs piled up until the server came back or the
plugin restarted.

The fix does three things:

1. The temp root is now mounted `soft` with `retry=0`. It only does
metadata work that a retried publish redoes, so an error there is
harmless, and a stuck call now ends on its own once the connection to
the server is declared dead. The mount the pod gets stays hard, because
a soft mount can report a failed write that the server did apply.
2. A publish to a target path that already has one running now returns
`Aborted` instead of starting another. This is what stops the pile-up:
at most one call per target path is stuck at a time.
3. Moving old files into `/data` now checks the destination with `lstat`
and fails the publish on any error other than a missing file. With the
soft mount, the check can fail with EIO on a file that `/data` already
holds, and a rename issued after the server comes back would replace
that file. `lstat` keeps a user's symlink in `/data` that loops or
dangles from failing every publish. Such a symlink counts as present and
is left alone, while before this change the rename replaced a dangling
one.

I kept the default retry timeouts on purpose, although the issue
suggested lowering `timeo` and `retrans`. The Linux client shares one
connection per server for all of that server's mounts on the node, and
the first mount sets the connection's TCP timeouts. Lowering them on the
temp mount would have changed the timeouts of the pods' hard mounts too.
Over NFSv4, a soft RPC fails only after the connection is declared dead,
so a stuck call takes minutes to end, longer than kubelet's 2m deadline.
The guard keeps kubelet's retries from adding new calls during that
time.

The guard works per target path, not per volume. Two pods on one node
that use the same RWX volume still publish in parallel, and each pod can
have at most one stuck call.

The unit tests pin the mount options on both mounts, the guard behaviour
including its release after a failed publish, and the destination check:
a failed check stops the move, and a symlink in `/data` stays in place.
They use the fake mounter. I did not test against a real NFS server that
went away.

Not covered here, and tracked in #4814:

- If the server's TCP connection stays up while the NFS service hangs, a
soft NFSv4 RPC does not time out. The guard still limits it to one stuck
call per target.
- `Mount` from mount-utils runs `mount` with no deadline. This predates
this change, and the guard bounds it the same way.
- If the target is already hard-mounted from the dead server, the check
of the target at the start of a publish blocks for good. The guard keeps
this to one call per target but does not end it. Unpublish is not
coordinated with a publish that is still running either. Both predate
this change.
- A follow-up comment on the issue reported two leftovers of the bounded
unmount: an orphaned `umount.nfs` after a timeout, and a possible `stat`
before `umount2`. Both may end sooner now that the temp mount is soft,
but I have not verified either.

Refs #4581

### Screenshots

Not a UI change.

### Downstream repositories

The change is limited to the kubevirt-csi node plugin's Go code. None of
the triggers in the contributing guide's map apply: no package added or
renamed, no platform values, variants, release assets,
ApplicationDefinition semantics or developer tooling touched.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kubernetes): an NFS volume publish in tenant Kubernetes clusters no longer hangs forever when the NFS server goes away during the publish, and kubelet retries no longer pile up while a publish is stuck. The plugin's temporary mount of the export root is now soft, and a second publish to the same target path is refused while the first is still running.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Concurrent NFS publish requests for the same target now return an
`Aborted` error. Different targets remain available, and a target can be
used again after a request finishes, including after a failed attempt.
* Migration now stops if checking a destination encounters an unexpected
error. Existing destinations, including symlinks, are left in place.
* Temporary-directory creation and mount errors now identify the
affected volume.
* **Updates**
* Temporary NFS root mounts now use soft behavior with retries disabled.
The final data mount remains hard.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-12 17:23:47 +03:00
Ray Low 904bc38fc3 fix(kubernetes): delete the Talos PKI and oidc-bootstrap Job on uninstall
cert-manager runs with enableCertificateOwnerRef: false, so the
<release>-talos-ca and <release>-talos-tls-cert Secrets carry no
ownerReference to their Certificates and outlive the release, keeping
the CA private key of a cluster that no longer exists. A new cluster
created under the same name then adopts that stale CA instead of
issuing its own trust root.

The <release>-oidc-bootstrap Job is a hook whose only delete policy is
before-hook-creation. Hooks are not part of the release manifest, so
uninstall never removes it.

The pre-delete hook deletes the two Certificates before their Secrets,
because deleting a Secret while its Certificate exists makes
cert-manager re-issue it. It runs after the TenantControlPlane is gone,
since the control plane's CSR signer mounts the CA. The Role grants
each delete by resourceName only.

Assisted-by: LLM
Signed-off-by: Ray Low <crossrt@gmail.com>
2026-10-11 22:40:47 +08:00
Aleksei Sviridkin bcde187858 fix(dashboard): small Console fixes from the #3828 triage (#3837)
## What this PR does

Six small Console fixes from the #3828 triage. Each one is small and
most of the code sits in `apps/console/src`, so they go together in one
PR.

`Fixes #3105`: sizes between 1Ki and 1Mi printed as a raw byte count.
`humanizeBytes` had branches for Ti, Gi and Mi and then fell through to
bytes. It now has a Ki branch with the same `toFixed(0)` as the Mi
branch. The `1023B` case still holds, and the test covers 1Ki and 512Ki.

`Fixes #3106`: the tenant picker showed on admin pages where the tenant
changes nothing. It is now hidden on the cluster-scoped admin pages
only: Capacity, Backup classes, External IPs and Modules. The rest of
`/admin` mounts the same tenant-scoped pages as `/console`, and those
take their namespace from the tenant context, so the picker stays there.
`/admin/tenants` keeps it too, because its rows come from the
picker-filtered tenant list.

`Fixes #3107`: both capacity drill-downs showed one generic error, so a
permission failure looked like a broken page. Both now use the `error
instanceof K8sApiError && error.status === 403` check that
`ClusterStorageSection` already uses, with a test for the forbidden case
and one for a generic error on each page.

`Fixes #3102`: an immutable leaf was not restored when the YAML edit
dropped its parent object. `overlayPath` stopped as soon as the
submitted body had nothing at a path segment. It now creates `{}` for
the missing parent, but only when the target there is undefined or null
and the original really has the leaf to restore. So a scalar or an
explicit null the user put there survives. A wildcard with more path
under it does not rebuild a deleted parent, since that would put back
entries the user removed. The test uses foundationdb's
`storage.storageClass`, which is a shipped path of this shape.

`Fixes #3135`: a blocked submit did not scroll to the field that blocked
it. Plain `focusOnFirstError` crashes here, because RJSF's built-in
handler reads `form.elements` and this form is `tagName="div"`. So the
form passes a small handler that finds the field by its generated id and
scrolls it into view. The ids come from one shared definition, and the
widgets that had none now carry it: the `DynamicOptionsWidget` select,
which every `x-cozystack-options` field uses, and the `SourceField`
controls. The nested form that `AdditionalPropertiesField` mounts per
map entry continues the outer id namespace instead of restarting at
`root`, so fields like `nodeGroups.*.instanceType` resolve. A map key
with a dot in it, such as a mariadb user `john.doe`, resolves by trying
both readings of the dotted error path.

`Fixes #3108`: `cilium`, `coredns` and `verticalPodAutoscaler` have only
`valuesOverride` and no `enabled`, so they rendered as plain groups next
to the toggleable addons and looked like a switch that failed to appear.
Setting `<addon>.enabled: true` in the YAML editor is accepted, because
the schema sets no `additionalProperties`, and the API stores the field
and echoes it back. Nothing reads it. The three addons are not the same
case. `cilium` and `coredns` are always installed.
`verticalPodAutoscaler` has no switch of its own and is installed and
removed together with `addons.monitoringAgents.enabled`. The three are
now marked in `values.yaml` with a new `x-cozystack-no-enable-switch`
schema keyword, and the form renders them with a note that there is no
enable switch and an `enabled` field in YAML does nothing. The per-addon
reason goes into the schema description, which the form renders and
which regenerates into the README, the Go types and the
ApplicationDefinition.

The generated files are regenerated with cozyvalues-gen v1.7.0, the
version CI pins, and the pre-commit check passes. The console typecheck
and tests pass in CI.

### Screenshots

##### Ki formatting
<img width="1182" height="184" alt="image"
src="https://github.com/user-attachments/assets/62662265-4f77-455d-b90e-85e417684bb9"
/>

##### Admin without tenant picker
<img width="1440" height="245" alt="image"
src="https://github.com/user-attachments/assets/99aff4f3-fda9-4c4c-8cf8-0c1eb01fb973"
/>

##### Capacity 403 error
<img width="1232" height="270" alt="image"
src="https://github.com/user-attachments/assets/f4e4b328-ee5e-4516-b78f-9b5084b6bbc8"
/>
<img width="1232" height="944" alt="image"
src="https://github.com/user-attachments/assets/142e438e-f538-436e-aae8-dfb79ec747e8"
/>

##### Validation focus
<img width="1166" height="135" alt="image"
src="https://github.com/user-attachments/assets/89abb73b-a624-406f-9736-3bf38bb8a923"
/>

##### Addon lifecycle
<img width="1166" height="591" alt="image"
src="https://github.com/user-attachments/assets/14a95891-84d1-4073-a009-bc0eab94f114"
/>

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against the
diff, file by file. This PR changes field descriptions, adds the
`x-cozystack-no-enable-switch` schema keyword on three addon objects,
and changes the console. No field is added, removed or renamed, and no
default, version enum, `kind`, `plural`, `release.prefix`, output Secret
or Service changes, so none of the terraform provider triggers fire. No
package is added or removed, and nothing changes in
`packages/core/platform/values.yaml`, the variants, `hack/` or the node
prerequisites. cozyvalues-gen itself is unchanged, since the keyword
goes through its generic `@x-*` passthrough. The kubernetes reference
page on the website regenerates from `README.md` on a stable tag, which
is automated rather than a follow-up.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(dashboard): Ki-range sizes now render as Ki instead of raw bytes, the tenant picker is hidden on cluster-scoped admin pages, capacity drill-downs distinguish a permission error from a broken page, an immutable field whose parent object is absent is now applied, a blocked submit scrolls to the field that blocked it including fields inside maps and the source and dropdown widgets, and the cluster addons with no enable switch now say so instead of showing a toggle that never appears
```
2026-10-10 23:47:46 +03:00
Aleksei Sviridkin f76dedf9ca fix(kubernetes): keep NFS publish from hanging on a dead server
The kubevirt-csi node plugin mounts the NFS export root into a temp
dir to create /data and migrate old files. That mount was hard, so
with the server gone the mkdir, readdir and rename calls on it never
returned. kubelet gives up after 2m and retries, and every retry
blocked the same way, piling up goroutines and nfs-init-* dirs.

Mount the temp root soft, so its RPCs fail once the connection to the
server is declared dead instead of blocking forever; it only does
metadata work that a retried publish redoes. retry=0 keeps mount.nfs
from retrying a failed temp mount for 2m. timeo and retrans keep their
defaults, because the first mount of a server sets them for the
connection the pods' mounts of it share. The volume mount the pod gets
stays hard, since a soft mount can report a failed write the server
applied.

Over NFSv4 that soft failure takes minutes, and a stat of a target
that is already hard-mounted can block for good, so a publish for a
target path that already has one running now returns Aborted instead
of starting another.

A soft stat can fail with EIO on a file /data already holds, and a
rename issued once the server is back would replace it, so the
migration now fails the publish on any stat error other than a
missing destination. It checks the destination with lstat, so a
user's symlink in /data that loops or dangles counts as present
instead of failing every publish or being replaced by the rename.

The skip it makes for a destination already in /data leaves the root
entry where the pod cannot see it, so it is now logged as a warning.
/data is created with a plain mkdir, since with the server gone
MkdirAll fails its stat on the mount point and then reports the local
temp dir as already existing instead of the EIO. The temp mount
errors now name the volume, like the migration errors do.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 20:46:12 +03:00
Aleksei Sviridkin 5486d4ffcf fix(networking): harden webhook policy assumptions (#4058)
## What this PR does

This is follow-up for
[#3997](https://github.com/cozystack/cozystack/pull/3997). It reads
ingress-nginx admission port from effective subchart values and refuses
empty value, and fixes `kubeProxyReplacement=false` status because
cozystack has no supported Service translator in that state. Gateway
test now requires one real ClusterIP and keeps same ready Pod UID, IP
and port for whole probe, so stale EndpointSlice, replaced Pod or wrong
Service can't report deny. Policy selectors and two-ended deny from
[#3997](https://github.com/cozystack/cozystack/pull/3997) stay
unchanged, and this does not claim root cause for
[#3983](https://github.com/cozystack/cozystack/issues/3983).

### Screenshots

Not applicable.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(networking): use effective ingress admission port, report unsupported Cilium service routing correctly, and reject stale backend evidence in gateway tests
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Admission webhook connectivity checks now validate the configured
Service and its exact backend endpoint throughout probing, improving
failure detection.
- Admission webhook policies now fail closed when the webhook port is
not explicitly configured.

- **Documentation**
- Clarified that supported service routing requires
`kubeProxyReplacement: true`.
- Updated guidance to explain that disabling this setting or webhook
policies does not provide a supported fallback.

- **Tests**
- Added coverage for missing webhook port configuration and strengthened
networking stability checks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 08:37:53 +03:00
IvanHunters ac4ab1c4a6 chore: drop private cluster and host names from the tree (#4461)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

I remove internal cluster names, a real hostname with its IP addresses,
internal mirror hostnames and an internal product name from comments, an
example file, test fixtures and a dead commented-out config block. Where
a stand-in is needed I use neutral test-cluster wording, RFC 2606
domains or RFC 5737 documentation addresses. I also replace two real
Keycloak usernames and a reviewer's name in code comments with
placeholders, or drop them. Rendered output does not change. The test
fixture hostnames change on the input and the expected side together,
and the unit tests of every touched package still pass.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Updated backup, OIDC, autoscaling, and platform references to describe
test environments without naming specific development clusters.
* Replaced development-specific endpoints in the platform example with
`example.org` and documentation-only IP addresses.
  * Updated OIDC username examples.
* **Tests**
  * Updated fixture hostnames and expected URLs to use `example.org`.
* **Chores**
* Removed commented-out registry mirror entries; the active mirror
configuration is unchanged.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 08:05:19 +03:00
Myasnikov Daniil 9a59f1d0f7 fix(dashboard, kubernetes): declare the no-enable-switch addons and harden the focus helper
Use the shared RJSF ID configuration in the focus helper, preserve
quoted dotted property names, and restrict its fallback to form inputs.
Add route and scalar-ancestor regression coverage for the rebased admin
hierarchy and immutable overlay.

Mark Kubernetes addons without their own switch explicitly in the
generated schema instead of inferring the behavior from their current
fields, and keep lifecycle-specific copy in each field description.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:04:06 +03:00
Myasnikov Daniil 35abe2dcbd fix(dashboard, kubernetes): say which cluster addons have no enable switch
cilium, coredns and verticalPodAutoscaler carry only valuesOverride and
no enabled, and the addon template keys its toggle off the presence of
enabled, so all three rendered as plain groups among the toggleable
addons, reading as a switch that failed to appear. That is what leads
users to add <addon>.enabled: true in the YAML editor, where the schema
sets no additionalProperties, so the API stores the field and echoes it
back while nothing reads it.

The three are not the same case, so the form must not claim they are.
cilium and coredns are always installed. verticalPodAutoscaler has no
switch of its own but is installed and removed together with
addons.monitoringAgents.enabled, so calling it mandatory would be
wrong. The fixed copy states only what holds for all three, that there
is no enable switch and that an enabled field in YAML does nothing, and
the per-addon reason moves into the schema description, which the form
already renders and which regenerates into the README, the Go types and
the ApplicationDefinition.

Also types the test fixture instead of spreading it as object, which
erased the props and left the file failing tsc.

Fixes #3108

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:03:51 +03:00
Myasnikov Daniil c9558f5658 fix(networking): correct webhook policy assumptions
Read the ingress admission port from the effective subchart values and fail if the port is absent. Describe kube-proxy replacement as a required service-routing invariant while retaining target-side enforcement.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:00:03 +03:00
Aleksei Sviridkin 92bf90cc7d fix(kubernetes): force the kubevirt-csi temp NFS unmount after a timeout
The NFS publish path temp-mounts the export root and unmounts it in a
defer with a plain umount, which has no deadline. If the NFS server is
gone by then, that umount can block indefinitely: the publish hangs and
the temp mount stays behind for the life of the node plugin container.

mount-utils' UnmountWithForce cannot fix this. It retries with -f only
after its first umount returns, and that umount forks the umount.nfs
helper: killed on the timeout, umount leaves the helper stuck on the
vanished server with the output pipe still open, and CombinedOutput
waits for the pipe, so -f never runs. The plugin therefore runs umount
itself with WaitDelay set, which closes the pipe shortly after the
kill, and falls back to umount -f under a timeout of its own. As in
mount-utils, "not mounted" counts as done: umount2 detaches the mount
before it blocks tearing the superblock down, so the -f after a killed
umount can find nothing left. A mounter that is not the exec-based one
keeps its plain Unmount.

An umount in uninterruptible sleep outlives any kill, so the cleanup
also waits at most one fixed budget, well inside the 2m deadline
kubelet puts on NodePublishVolume, and leaves a stalled unmount
running in the background, where it logs its result once it ends. The
temp dir is removed by that background unmount once it succeeds, not
by the publish: removed at the deadline it is still a mount point, and
nothing would retry.

A forced unmount runs nfs_umount_begin, which kills every RPC in
flight on the superblock, and NFS shares one superblock between mounts
of the same export with the same options. The temp mount therefore
gets nosharecache, so -f cannot fail the I/O of pods on the node that
mount the same volume.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-07 04:38:53 +03:00
Aleksei Sviridkin 520af57905 chore: drop private cluster, host and personal names from the tree
Private infrastructure names, a real hostname, a commented-out
internal mirror block, an internal product name, and a reviewer's
name in a comment do not belong in a public repository; drop them
and replace what needs a stand-in with neutral placeholders.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-07 04:38:06 +03:00
Aleksei Sviridkin d2e3eb4f82 fix(kubernetes, e2e): unblock autoscaling below min size, size the computeplane pool, fix the backupcontroller build (#4768)
## What this PR does

Makes the backupcontroller test package compile again on main. #4587
added a flavor argument to `createCNPGBackupArtifact` and
`marshalCNPGBackupSnapshot`, and #4669, whose CI ran before that, added
two test calls with the old argument lists. Both PRs were green on their
own, and together they break `go test ./internal/backupcontroller/`. The
two calls now pass the same default flavor as the other tests in the
file.

The second commit fixes an e2e suite that fails the required E2E check
here. It is the `computeplane` suite from #4713, and its worker pool was
one node with a single vCPU. After the kubelet reservations that node
has 900m of CPU to allocate. The tenant addons (cilium, coredns, csi,
ingress-nginx, metrics-server, cert-manager) request exactly 900m
between them. If cert-manager's startupapicheck hook pod is created
last, it can't be scheduled and the install times out after 10m. The
reinstall after remediation runs into the same full node. Whether that
pod comes last depends on install order, so the suite passed on #4713
and failed here.

The pool now has two workers with one vCPU each, the same shape the
kubernetes suites use. One vCPU per worker stays on purpose, because of
#3513.

Two workers exposed a deadlock in the tenant cluster-autoscaler. The
suite's pool is created at zero replicas and then gets a min size of 2,
which CAPI does not apply to the live replica count on upgrade (#4786).
The autoscaler's scale-up for a pending pod asks for one node, and the
clusterapi provider refuses it because one is under the min size,
although it is an increase. That refusal backs the group off, and the
min-size enforcement skips groups in backoff, so the pool never leaves
zero. Upstream had the same report in kubernetes/autoscaler#4942, closed
as stale, and it is now tracked in kubernetes/autoscaler#10400. Our
autoscaler image already patches the mirror case for max size, and the
patch now refuses only a decrease below the min size.

`go vet` and `go test ./internal/backupcontroller/...` pass, and so do
the bats pins for the computeplane suite. The autoscaler patch adds a
test for each direction, and the clusterapi package tests pass with it
applied to 1.32.0.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved cluster autoscaling for worker groups outside configured size
limits: groups below their minimum can scale up toward it, and nodes can
be removed when a group exceeds its maximum. Scale-down requests below
the minimum remain restricted.
* **Tests**
* Updated backup and restore test calls with PostgreSQL and empty flavor
values.
* Updated ComputePlane checks for a two-worker pool and clarified
cert-manager installation timing expectations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 04:36:10 +03:00
Aleksei Sviridkin 82ae9f6cf7 fix(kubernetes): let the autoscaler grow a node group that sits below min
The clusterapi provider refuses any SetSize under the group's min size,
including an increase. A pool whose min size is raised while it stands
at fewer replicas then deadlocks: the scale-up for pending pods asks for
fewer nodes than the min, is refused, and puts the group in backoff,
and the min-size enforcement skips groups in backoff. The group stays
where it is for good.

Refuse only a decrease below min size. The guard it keeps is the one
DecreaseTargetSize relies on, and DeleteNodes checks min size itself.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-07 00:58:11 +03:00
Aleksei Sviridkin bcda5c893a feat(kubevirt)!: drop HotplugVolumes for declarative volume hotplug (#4453)
## What this PR does

This turns off the deprecated KubeVirt `HotplugVolumes` gate and moves
the platform to declarative volume hotplug. It changes how every volume
hotplug on the platform works, so the title carries `!`. Closes #4032.

This PR is stacked on #4451 and has to merge after it. #4451 adds the
kubevirt-csi flag this PR flips.

The replacement gate is already on in KubeVirt 1.9, and the old gate is
the only thing keeping it off. `DeclarativeHotplugVolumes` is Beta at
v1.9.0, and Beta gates are on by default. virt-controller skips
`handleDeclarativeVolumeHotplug` while `HotplugVolumes` is on, so
dropping the old gate is what switches the new one on.

The kubevirt-csi fallback goes off in the same change. With the old gate
off, virt-api rejects the VMI-level `removevolume` on a VM-owned VMI, so
the chart now passes `--enable-vmi-hotplug-fallback=false`. The comments
next to the gate and the flag point at each other.

This needs the rebuilt kubevirt-csi image. A driver built before #4451
does not know the flag and exits on it at startup. The image is stamped
at release preparation and ships in the same release as this change, so
only a `make apply` from main before then would crash-loop the CSI
controller.

What changes at runtime on an upgrade:

- The VM-level `addvolume` and `removevolume` calls now edit `VM.spec`
directly. virt-controller then makes the VMI hotplug volumes match it,
instead of going through `VM.status.volumeRequests`.
- Volumes hot-plugged through the VM subresource are in `VM.spec`, so
they stay attached.
- During a tenant teardown, when the VM goes before its VMI, the driver
no longer calls the VMI subresource. It waits for the VMI status to drop
the volume, and the garbage collector deletes the VMI with its VM.
Upstream waits up to two minutes. If the VMI is still there after that,
the unpublish fails and is retried.

A new platform migration keeps old VMI-only volumes attached. On its
first pass virt-controller removes every hotpluggable VMI volume that is
missing from `VM.spec`. Until July 2025 kubevirt-csi attached volumes at
the VMI level (the Role then granted
`virtualmachineinstances/addvolume`), and live migration carries such a
VMI across KubeVirt updates. Without help, a tenant node not recreated
since then would lose a mounted disk on upgrade.

Migration 59 copies each such volume and its disk from the VMI into the
VM template before anything else changes. Adding a hotplug volume to the
VM spec is a valid live update in KubeVirt, so no VM needs a restart. It
runs as the platform chart's pre-upgrade hook, before the upgrade
re-points `cozystack-packages`. The KubeVirt CR and the tenant
kubernetes charts both render from that source, so the gate and the flag
cannot change before the migration has finished. It only touches VMs
that need it, logs each one, and a second run changes nothing.

The migration also makes ephemeral hotplugs permanent. A volume a user
added at the VMI level, for example with `virtctl addvolume` without
`--persist` from an older virtctl, has the same shape as kubevirt-csi's
old attachments, because virt-api builds both the same way. There is
nothing reliable to tell them apart by, so both are copied. Leaving the
user's volume out would not keep it ephemeral: the first reconcile would
unplug it from the running VM. After the upgrade it is part of `VM.spec`
and attaches again on a restart, and `virtctl removevolume` drops it.

Ephemeral hotplug stops working on VMs. With the old gate off, virt-api
refuses VMI-level `addvolume` and `removevolume` on a VMI that a VM owns
(`vmiVolumePatch` in v1.9.0). A volume on such a VMI is added and
removed through the VM from then on. A VMI without a VM is not affected.

Only running VMIs are copied. virt-controller reconciles no other VMI,
and a stopped VM builds its next VMI from `VM.spec`, so copying from a
stopped one would bring back volumes the restart was going to drop.

Migrations run only on the isp-full, isp-full-generic and isp-hosted
variants. KubeVirt comes from the platform only on isp-full and
isp-full-generic, so those are covered. On the default variant, where
packages are installed by hand, nothing runs the migration.

This optional check lists the VMIs and volumes the migration would copy.
It uses the same rule. On the default variant, run it before upgrading
and recreate each tenant node it lists:

```bash
kubectl get vm,vmi --all-namespaces --output json | jq --raw-output '
  ([.items[] | select(.kind == "VirtualMachine")
    | {key: "\(.metadata.namespace)/\(.metadata.name)", value: [.spec.template.spec.volumes[]?.name]}]
    | from_entries) as $vm
  | .items[] | select(.kind == "VirtualMachineInstance")
  | ([.metadata.ownerReferences[]? | select(.controller == true and .kind == "VirtualMachine" and (.apiVersion | startswith("kubevirt.io/"))) | .name][0]) as $owner
  | "\(.metadata.namespace)/\($owner)" as $key
  | select(.status.phase == "Running" and $owner != null and $vm[$key] != null)
  | .spec.volumes[]? | select(.dataVolume.hotpluggable == true or .persistentVolumeClaim.hotpluggable == true)
  | .name as $n | select($vm[$key] | index($n) | not)
  | "\($key) \($n)"'
```

Empty output means nothing is affected. I checked the query against the
migration's test fixtures only. Neither the query nor the migration has
run on a live cluster.

#4287 also adds a migration 59. If it merges first, this one moves to
60: the file, the version it stamps, `migrations.targetVersion`, the
bats file (its `run_migration` argument and expected `STAMP`) and
`hack/testdata/migration-59-vmi-hotplug`.

A VM with a pending hotplug request at upgrade time may also get stuck.
virt-controller still processes `VM.status.volumeRequests` before the
declarative handler, and it applies them through the VMI subresource,
which virt-api rejects once the old gate is off. The sync then fails
before the VM spec is updated, so nothing clears the request, and every
later sync of that VM fails the same way. A retry through the VM
subresource edits the spec and clears it, which covers kubevirt-csi's
own requests. A one-shot request stays stuck until the VM restarts. I
read this from the v1.9.0 code and haven't reproduced it. Waiting until
this lists nothing before upgrading narrows the window, but a request
made during the rollout can still hit it:

```bash
kubectl get vm --all-namespaces --output json | jq --raw-output '.items[] | select((.status.volumeRequests // []) | length > 0) | .metadata.namespace + "/" + .metadata.name'
```

The two halves live in different HelmReleases, so there is a short
window during the rollout where the driver and the gate disagree. In
that window an unpublish fails and is retried until both are updated.

A tenant cluster whose `kubernetes` HelmRelease is suspended or failing
keeps its old driver for longer. Drivers from before July 2025 attach
through the VMI subresource, so on such a tenant every attach is refused
until its release reconciles. Such a driver can also attach a volume
after the migration has run and before the gate flips. The migration
doesn't see that volume, and the first declarative reconcile unplugs it
from the running VM.

The migration lists the VMs once and patches each one with a
`resourceVersion` test, so a VM that changes in between fails the hook
Job. The Job has `backoffLimit: 3`, and each retry lists again and skips
the VMs already patched. Re-reading only the conflicting VM would avoid
the retry; I left that out.

The new gate is listed in the CR even though it is on by default. With
the old gate gone it is the only hotplug path, and the chart refuses a
render that disables it. I found no other user of `HotplugVolumes` in
the platform or in the downstream repositories.

The VMI-level fallback may have been failing already. The kcsi Role has
no rule for the VMI-level `removevolume` (#4452), so that call likely
got 403 with or without the gate. I haven't checked that on a cluster.

`hack/migration-59-vmi-hotplug-volumes.bats` runs the migration against
a fake kubectl: a VM that needs the copy, VMs and VMIs that must be left
alone (a VMI that is not Running among them), a second run, a cluster
without KubeVirt and the fail-closed paths. The helm-unittests for the
KubeVirt CR pin the whole gate list, accept `HotplugVolumes` in
`disabledFeatureGates` and refuse `DeclarativeHotplugVolumes` there. The
CSI test pins the flag at false, and the `make update` idempotency test
still passes. None of that exercises hotplug itself. The real test is
e2e with volume attach and detach on a tenant cluster, including a
teardown. No CI lane upgrades from the previous release yet, so an
upgrade with a live VMI-only volume is untested; #4463 tracks that test.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kubevirt)!: KubeVirt now runs with DeclarativeHotplugVolumes instead of the deprecated HotplugVolumes feature gate, and kubevirt-csi no longer uses the VMI-level hot-unplug. Volume hotplug requests are applied through VM.spec. The upgrade copies volumes that older kubevirt-csi versions attached only to the VMI into VM.spec, so they stay attached; the check in the PR description is optional. This also applies to ephemeral volumes added to running user VMs at the VMI level, such as with virtctl addvolume without --persist: they become part of VM.spec and attach again after a VM restart. virt-api now refuses VMI-level addvolume and removevolume on a VMI owned by a VM, so such volumes are added and removed through the VM. A tenant cluster whose kubernetes release is suspended or failing keeps its old kubevirt-csi, and a driver from before July 2025 cannot attach volumes there until that release reconciles, and a volume it attaches during the upgrade itself can be unplugged again. A values override that puts DeclarativeHotplugVolumes into disabledFeatureGates now fails the kubevirt render. Clusters on the default variant do not run platform migrations and should run that check before upgrading.
```





<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
  * Declarative hotplug volumes are enabled by default for KubeVirt.
* Running virtual machines keep eligible hotplugged volumes and disks
when their configuration is migrated.
* **Bug Fixes**
* The migration avoids adding duplicate volumes or disks and does not
mark the migration complete if required operations fail.
  * The Kubernetes CSI driver no longer uses the VMI hotplug fallback.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 17:06:03 +03:00
IvanHunters 86bf2ba8ec fix(kubernetes): let kubevirt-csi hot-unplug from an orphaned VMI (#4563)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

This lets the kubevirt-csi controller detach a volume from a VM instance
that no VM owns. Right now that call is always denied. Closes #4452.

The driver has a fallback for a volume that is still attached to the VMI
but is no longer in the VM spec. It calls the VMI-level `removevolume`
subresource. The `-kcsi` Role grants only the two VM-level subresources,
so the call ends in 403. Nothing else reaches this service account with
that grant. Only its own Role and ClusterRole name it as a subject, and
the group bindings that cover every service account grant nothing on
this subresource. At v1.9.0, `kubevirt.io:default` grants only
`kubevirts`, `version` and `guestfs`. `kubevirt.io:edit` and
`kubevirt.io:admin` do include the VMI subresource, but they aggregate
only into the built-in `edit` and `admin` roles, and no tenant binding
gives those to this account. The cozystack tenant roles aggregate on
their own labels and don't list it.

The fallback is still needed after the flag in #4453 goes to false. The
flag only skips VMIs that a VM owns. When a VM is deleted with
`propagationPolicy=Orphan`, the KubeVirt VM controller drops its
finalizer and leaves the VMI running (`vm.go` at v1.9.0). The tenant
node stays up with its volumes attached. When a pod moves off that node,
the driver has to detach through the VMI, and on a 403 the hot-plug pod
keeps holding the infra volume. Until #4453 lands, the fallback also
runs for VM-owned VMIs and gets the same 403.

The grant is `update` on `virtualmachineinstances/removevolume` and
nothing more. The account can already remove a volume from any VM in the
namespace, so it gets no new reach. virt-api also rejects the VMI-level
call on a VM-owned VMI once `HotplugVolumes` is off.
`virtualmachineinstances/addvolume` stays out because the driver only
attaches through the VM.

The new helm-unittest pins the Role's `subresources.kubevirt.io` rule to
these three subresources with `update`. I haven't reproduced the 403 on
a cluster. This shows it on a tenant namespace, before and after:

```bash
kubectl auth can-i update virtualmachineinstances.subresources.kubevirt.io --subresource=removevolume \
  --namespace <tenant-ns> --as system:serviceaccount:<tenant-ns>:<release>-kcsi
```

Upstream kubevirt/csi-driver has the same gap in
`deploy/infra-cluster-service-account.yaml`.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubernetes): the kubevirt-csi controller can now hot-unplug a volume from a VMI that no VM owns, instead of failing with 403 Forbidden.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved Kubernetes CSI volume removal for virtual machine instances,
including cases where a volume is not listed in the virtual machine
specification. This helps the driver complete volume removal in
scenarios where the Kubernetes virtualization API accepts the
instance-level request. Existing permissions for adding and removing
volumes from virtual machines remain unchanged.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 15:14:19 +03:00
IvanHunters c7b8009964 fix(kubernetes): log kubevirt-csi temp NFS mount dir removal failure (#4562)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The kubevirt-csi node plugin now logs a warning when it cannot remove
the temp dir it uses for NFS publish. Before, the error was dropped, so
a dir left behind after a failed unmount was never reported.

The publish path temp-mounts the export root to create `/data`, then
removes the temp dir in a defer. The failure is logged, the same way the
unmount failure next to it already is, and not returned. A leftover temp
dir does not break the volume, and failing a publish that otherwise
worked only makes kubelet retry it.

A new unit test publishes an NFS volume through the fake mounter, where
the removal always fails, and checks that a warning naming the dir is
logged.

This is independent of #4451. That PR touches the same Go module but not
`node.go`.

A failed unmount still leaves the NFS root mounted. That is older than
this change and is tracked in #4561.

Closes #4466

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubernetes): the kubevirt-csi node plugin now logs a warning when it cannot remove its temporary NFS mount dir instead of dropping the error
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Failed cleanup of a temporary mount directory now produces a warning
with the directory path, making the issue easier to identify. The volume
publish operation continues to succeed when cleanup fails.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:39:58 +03:00
Aleksei Sviridkin 2af8b0c229 ci(build): fall back to direct, retry and cache Go module downloads (#4660)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

One transient error from proxy.golang.org is enough to fail an image
build or the `Unit & controller tests` job today. The error is an HTTP/2
stream reset (`stream error ... INTERNAL_ERROR`), and nothing retries or
falls back after it. This PR adds three things: a direct fallback, a
bounded retry and a module cache.

Image builds now fall back to fetching modules from their origin when
the proxy fails. They get `GOPROXY=https://proxy.golang.org|direct` as a
build arg from `hack/common-envs.mk`, and every Go builder stage
declares `ARG GOPROXY`. With a pipe Go goes direct on any proxy error.
With Go's default comma it does that only on a 404 or 410. The value
comes from `IMAGE_GOPROXY` and not from the host's `GOPROXY`, because
buildkit can't reach a mirror on localhost or a `file://` proxy. Every
workflow job that runs Go on the runner sets the same value as env.

Explicit module downloads in Dockerfiles are retried three times with a
growing pause. The loop is inline, in one form that the contract test
pins. I tried a shared script from a named build context first, but
buildx refuses any `--build-context` for the four Dockerfiles still
pinned to the `dockerfile:1.2` frontend. The images also build from
different contexts, so no single file can be copied in. Raising those
pins changes how those images build and needs a real build to check, so
I left it for a follow-up. Two alpine stages, token-proxy and
shadowbox's `ss-server`, left the download to `go build`. The alpine
golang images have no git, so the direct fallback can't fetch for them,
and they now get the same download step. The runner jobs use
`hack/go-mod-download.sh` with the same bound.

The unit test job now starts from a module cache instead of downloading
everything. It restores `GOMODCACHE` with `actions/cache`, keyed on
every `go.sum` in the tree, and then prefetches all tracked modules
through the retry, so a cache miss still works. A cache saved by a PR
run is visible only to that PR, but one saved on the default branch can
be restored by any PR
([docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#restrictions-for-accessing-a-cache)).
That is why the build cache warmer on main writes the same key before
its image build. This way the first run of a PR, which is the one that
flakes, starts warm. The warmer restores only an exact key, so old
module versions don't pile up, and its Go steps can't fail the image
build. Both runner labels are Linux, so the key matches. Both also check
out under `/home/ubuntu/_work`, so `go env GOMODCACHE` should give the
same cache path, but I have not seen it on the warmer's runner yet. The
release jobs in `tags.yaml` and `promote-rc.yaml` get the GOPROXY env
only, no cache.

`hack/go-module-fetch-contract.bats` covers all three. Each test is red
on main, and I checked each guard with a mutation.

Closes #4591

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
ci(build): Go module downloads in image builds and CI jobs now fall back to direct fetching and retry on transient proxy.golang.org errors. The unit test job reuses a module cache written from main.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Reliability**
- Go module downloads now fall back to direct access when the public
proxy encounters errors.
- Builds and automated checks retry failed module downloads, helping
them recover from temporary network issues.
- **Performance**
- CI caches Go modules to reduce repeated downloads and speed up builds
and tests.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:14:47 +03:00
IvanHunters 94abf6940b feat(kubernetes): let kubevirt-csi detach without the HotplugVolumes gate (#4451)
## What this PR does

This lets kubevirt-csi detach volumes without the deprecated KubeVirt
`HotplugVolumes` gate. The default stays as it is, so nothing changes at
runtime yet. Part of #4032.

The driver got a switch upstream for exactly this, and I bumped to it.
kubevirt/csi-driver#382 added `--enable-vmi-hotplug-fallback`, merged as
cb4770e4e475. With it set to false, the driver no longer calls the
VMI-level `removevolume` on a VMI owned by a VM. virt-api rejects that
call once `HotplugVolumes` is off.

Our wrapper now passes the flag to upstream and no longer has its own
hot-unplug path. `unpublishHotplugVolume` was a backport of
kubevirt/csi-driver#184, and the previous pin already contained that
change. Upstream detaches from the VMI when the VM is gone, then waits
for the VMI status to drop the volume before it returns. So the
wrapper's re-check never found the volume attached. Adding the ownership
check there would have been dead code.

Closes #4450

The chart leaves the flag at its default of true and does not render it.
The driver image in the chart is stamped only at release preparation, so
between releases main runs the previous release's binary, which does not
know the flag. Go's `flag` package exits on an unknown flag. Rendering
`=true` would crash-loop the CSI controller for no effect. The right
value depends on the KubeVirt CR, not on the tenant, so the comment next
to `HotplugVolumes` in `kubevirt-cr.yaml` points at the controller
Deployment. The follow-up PR drops the gate and renders the flag as
false.

Nothing changes on the node side. Main already pins f5de10d39556, and
cb4770e4e475 is that commit plus kubevirt/csi-driver#382.

A Go unit test runs an unpublish through the wrapper during a teardown,
with the flag on, off with a VM-owned VMI, and off with an orphaned VMI.
It checks that the flag reaches upstream and that only upstream makes
the VMI-level call. Upstream's own tests cover the ownership rule. A
second test holds that an NFS volume only releases the node from its
CiliumNetworkPolicy and never reaches the upstream hot-unplug. A
helm-unittest pins that the chart renders no such argument. I haven't
built the image or run e2e. Volume attach and detach on a tenant cluster
in e2e is the real check.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kubernetes): kubevirt-csi supports --enable-vmi-hotplug-fallback, so it can detach volumes without the KubeVirt HotplugVolumes feature gate. The flag stays true for now and behaviour is unchanged.
```
2026-10-05 10:46:12 +03:00
Aleksei Sviridkin b2a5f12e73 build(kubernetes): run the tests of the packages the patches touch
The cluster-autoscaler and kubevirt-cloud-provider image builds ran go
build only, and go build never compiles a _test.go, so nothing ran the
tests of the code the patches change. Run go test in the builder stage
on the packages the patches touch. The list is read from the patches
themselves, so a new patch is covered without an edit to the Dockerfile.

Both builder stages export GOOS and GOARCH for the target, so the tests
unset them: a local make image builds amd64 and arm64 by default, and a
test binary for the other architecture runs on the build host only
under binfmt emulation.

The first run of that step on cluster-autoscaler found the carried
downscale patch out of step with the upstream tests it leaves in place:
SetSize no longer refuses a size above maxSize, while one case of
TestNodeGroupIncreaseSizeErrors still expected that refusal. The patch
now drops that case, and carries a test for the downscale it is named
for instead. DeleteNodes lowers a group one node at a time through
SetSize, so with the refusal in place a group standing above its max
size could not shed a node at all. The new test deletes a node from
such a group and expects the replica count to drop; against the
unpatched SetSize it fails with the refusal the patch removes. Scale-up
past maxSize stays bounded by the core scale-up orchestrator, which
skips a node group already at MaxSize.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 22:01:01 +03:00
Aleksei Sviridkin 64f43f4f77 fix(computeplane): do not grant tenants the cluster admin kubeconfig
The computeplane module deploys the kubernetes chart into the tenant
namespace, and that chart's dashboard Role lets the tenant's "use"
subjects read <release>-admin-kubeconfig. Withholding the Secret from
the tenant secrets view does not help when RBAC grants it directly, so
any tenant user at "use" level or above could fetch the cluster-admin
credential of a cluster the module promises to keep operator-owned.

The kubernetes chart now drops that rule when the internal value
withholdAdminKubeconfig is set, and the module sets it in the cluster
HelmRelease's inline values. The tenant cannot reach those values: the
module reads only nodeGroups from them and splices an allowlisted
subset into the pool releases, and Flux applies spec.values over
valuesFrom. A regular kubernetes app keeps the rule, so its dashboard
can still offer the kubeconfig.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:36:57 +03:00
Aleksei Sviridkin fccb7fee06 test(tests): fail absence-only cases when the template renders nothing
Each of these templates is wrapped in a top-level condition, and the
case asserts only that a field is absent. If a values change turns the
condition off, the render is empty and the absence holds trivially, so
the case passes without looking at anything. A document selector makes
an empty render fail with "document not found".

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 22:27:17 +03:00
Aleksei Sviridkin 548d33f01f build(images): fall back to direct and retry Go module downloads
A transient proxy.golang.org HTTP/2 stream reset fails an image build
outright: Go's default GOPROXY falls back to direct only on a 404 or
410. Image builds now get GOPROXY=https://proxy.golang.org|direct as a
build arg from hack/common-envs.mk, declared by every Go builder stage,
so any proxy error falls back to fetching from the origin. The value
comes from IMAGE_GOPROXY rather than the host's GOPROXY, which may
name a mirror that buildkit cannot reach. The alpine golang images
carry no git, so in their stages a module hosted on GitHub cannot be
fetched directly and the retry is the cover.

Explicit `go mod download` steps retry three times with a growing
pause. The loop is inline, in one form the contract test pins: the
images build from different contexts, so no shared script can be
copied in, and a named build context in the shared buildx arguments
makes buildx refuse every Dockerfile pinned to a frontend older than
dockerfile:1.4.

The two alpine stages that left the download to `go build` get the
same download step, because without git neither the fallback nor a
retry covered them.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 19:37:00 +03:00
Aleksei Sviridkin d6b38f0a79 fix(kubernetes): fail the OIDC bootstrap Job when the tenant CA is missing (#4232)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

In `spec.oidc.mode: System`, the post-install/post-upgrade bootstrap Job
patches the chart-owned `<release>-oidc-kubeconfig` Secret with a real
kubeconfig built from the tenant admin kubeconfig's CA data. When that
CA data was empty, the script logged a WARNING and continued, then
exited 0. The hook was satisfied, the HelmRelease reported `Ready`, and
the Secret was left holding the install-time placeholder kubeconfig,
which cannot authenticate. This PR makes the Job fail (`exit 1`)
instead, so the failure surfaces as a red install and can be retried
(the Job already has `backoffLimit: 6` and `restartPolicy: OnFailure`)
rather than reported as a falsely healthy release. This is the smallest
of the three directions raised on the linked issue, and matches how the
neighbouring failure mode in the same script already behaves (a failed
`kubectl apply` of the ClusterRoleBindings already fails the Job under
`set -euo pipefail`). Behaviour for `mode: None` and `mode:
CustomConfig` is unchanged, and the common case — CA data present — runs
the identical patch commands as before.

A regression test was added to
`packages/apps/kubernetes/tests/oidc_test.yaml` asserting the rendered
Job script contains the `exit 1` on empty `CA_DATA` and no longer
contains the old warn-and-continue text. Confirmed it fails against the
pre-fix template and passes against the fix.

Validation:
- `helm unittest .` in `packages/apps/kubernetes` — 150/150 tests pass
(up from 149 before the new test).
- `helm template ... --show-only templates/oidc-rbac-job.yaml` with
representative `System`-mode values renders valid YAML; the extracted
bootstrap script passes `bash -n` and `shellcheck -s sh` introduces no
new warnings versus the pre-change script.
- `make generate` was not run/needed — this change touches neither
`values.yaml`, `values.schema.json`, `Chart.yaml`, nor `README.md`.
- Not run: a live cluster reproduction (Kamaji-issued admin kubeconfig
actually missing CA data at Job run time). The defect is provable from
the script logic itself and the added unit test demonstrates the
fail-then-pass behaviour; there is no live cluster available in this
environment.

### Screenshots

Not applicable — no UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

This change is confined to the shell logic inside
`packages/apps/kubernetes/templates/oidc-rbac-job.yaml`'s post-install
hook Job (a `Values.oidc.mode: System`-only failure path) and its
accompanying helm-unittest test. It does not add, rename or remove a
package, a `values.yaml` key, a CRD, a namespace, a release-owned object
name, or any other surface the trigger map lists, so no downstream
repository is affected.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubernetes): a System-mode OIDC install no longer reports Ready while the tenant OIDC kubeconfig Secret still holds the install-time placeholder — the bootstrap Job now fails (and retries) instead of only logging a warning when the tenant cluster's CA data isn't available yet.
```

Fixes #3765


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* OIDC setup now fails with an error if the tenant admin kubeconfig is
missing certificate authority data, rather than leaving the OIDC
kubeconfig Secret at its install-time placeholder.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-30 01:50:22 +03:00
Pujitha Paladugu 4fe45eba32 fix(kubernetes): fail the OIDC bootstrap Job when the tenant CA is missing
In mode=System the post-install hook Job patches the chart-owned
<release>-oidc-kubeconfig Secret with a real kubeconfig built from the
tenant admin kubeconfig's CA data. When that CA data was empty the
script only warned and exited 0, so the hook was satisfied, the
HelmRelease reported Ready, and the Secret kept the install-time
placeholder kubeconfig, which cannot authenticate. The failure only
showed up when a user tried to use the downloaded kubeconfig.

Exit 1 instead, so the install fails visibly and is retried. The Job
already has backoffLimit: 6 and restartPolicy: OnFailure, and a failed
ClusterRoleBinding apply in the same script already fails the Job the
same way.

Assisted-by: LLM
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
2026-09-30 00:36:44 +03:00
Marian Koreniuk 5be222ad75 docs(kubernetes): say the token's user needs the pool privileges too
A privilege-separated Proxmox token never exceeds its user's permissions.
Granting the README's roles to the token alone left the CSI and CCM tokens
with an empty effective set on the pool and the storage; granting the same
roles to the user made the token's own ACL the effective one. Seen on
Proxmox VE 9.2.20 while running the storage pair against a pool-scoped
user, and worth one sentence next to the table so nobody repeats it.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
2026-09-29 21:04:06 +02:00
Marian Koreniuk 36b04218d7 test(kubernetes): assert where the hypervisor token is and where it is not
Every assert here was written against the mutation it exists for, and each was
checked to turn red on it and green without: putting the controller back in the
tenant, giving the tenant a non-empty config.clusters, pointing a sidecar at the
wrong kubeconfig, restoring the blanket toleration, hardcoding the driver's
default owner id, dropping capmox mode, and letting the controller ignore
proxmox.csi.enabled.

The negations carry `any: true`. Without it a negated containsDocument passes
whether or not the document is in the render.

Two of the cases are about objects rather than values: the tenant release must
carry no valuesFrom at all, and two clusters must derive different volume owner
ids. Both are properties a reader can check against the rendered output, which
is why they are asserted that way rather than by reading the templates.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-29 21:04:05 +02:00
Marian Koreniuk 5ab36023e0 fix(kubernetes): release Proxmox volumes before the driver that frees them goes
Deleting a tenant cluster left its PVC disks on the hypervisor. The pre-delete
hook deletes every HelmRelease labelled with the cluster in Step 2, the CSI
driver among them, and nothing before that deleted the tenant's PVCs — a volume
is freed only by the driver answering DeleteVolume. capmox then destroys the
worker VMs, and Proxmox frees only what the destroyed VMID owns, which these
volumes are not. They stayed on storage with the tenant's data in them, and the
PV objects that could have named their owner went with the control plane.

Step 1b deletes the claims while the driver is still up, and it deletes the
claims before the Pods that hold them: a claim being deleted is one the
scheduler places no new Pod on, so a Pod that a StatefulSet or Deployment brings
back cannot mount it again and hold it to the backstop. The Pods go second: a
claim in use carries the kubernetes.io/pvc-protection finalizer, so it clears
only once nothing mounts it, and the Pods are going with the cluster in any
case. Then it waits for the PVs rather than the PVCs, because the claim
disappears as soon as its finalizer clears while the disk is freed only when the
PV is gone.

The Pods holding a claim are found by listing every Pod with the claims it
mounts and matching in the shell. A JSONPath filter on the claim name cannot do
it: with a Pod that mounts two claims its left side yields two values, client-go
refuses the whole expression ("can only compare one element at a time"), and no
Pod in that namespace would be found for any claim.

A kubeconfig that exists is not an apiserver that answers, and listing claims
against one that does not comes back empty — the clean case. Step 1b asks
/readyz first and sends an apiserver that stays silent through the backstop, so
the summary carries it. The probe is bounded by `timeout` rather than by
--request-timeout: a non-zero --request-timeout on an in-cluster kubectl call
diverts it to localhost:8080, which is why hack/check-hook-request-timeout.bats
refuses the flag in hook templates, and the check cannot tell this call, which
carries its own kubeconfig, from one that does not.

Best effort, and loud about it: a tenant that never came up has no apiserver to
ask, and a teardown must not wedge on that, but a timeout is reported through
the same backstop the other steps use.

The hook's Role gains one Secret by name — the tenant's admin kubeconfig, which
this step needs. Without it the step takes its no-kubeconfig branch and reports
that there is nothing to release while the volumes stay behind.

Step 5 no longer fails the whole hook where CDI is not installed. It deletes
DataVolumes, which are a KubeVirt-path component, and kubectl exits non-zero on
an unknown resource type; under set -e that failed the teardown after every
earlier step had already run, leaving the cluster half removed and the hook
reporting failure.

Verified on a throwaway tenant: a PVC bound and mounted by a running Pod, then
`helm uninstall`, and the volume freed by the driver rather than by hand.

delete_hook_test.yaml pins Step 1b and the Role's admin-kubeconfig entry on
proxmox and their absence on kubevirt, the /readyz probe, and the order of the
claim and Pod deletes.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-29 21:04:05 +02:00
Marian Koreniuk d902421f47 feat(kubernetes): deploy the Proxmox cloud-controller-manager with the cluster
On this substrate the worker kubelet runs with --cloud-provider=external, which
taints every node node.cloudprovider.kubernetes.io/uninitialized until a cloud
provider clears it. Without a cloud-controller-manager the taint is permanent:
nothing but tolerating DaemonSets schedules, and the CSI node plugin — which
reads topology.kubernetes.io/region and zone off its own Node and exits fatally
without them — never starts.

It runs in the management cluster for the same reason as the CSI controller
beside it: it authenticates to the hypervisor, and the tenant is cluster-admin
of its own cluster. Its whole job is to write on the tenant's Nodes, and it
does that over the tenant's admin kubeconfig from out of cluster, which also
means there is no tenant-side RBAC left to install.

capmox mode, not the default one. The default resolves a node to its VM by name
and writes providerID as proxmox://<node>/<vmid>, while capmox reads the CAPI
Machine and writes proxmox://<SMBIOS UUID> — the form capmox itself puts on the
Machine. With the default the Node and its Machine carry two different
identifiers for the same VM.

The tolerations are spelled out rather than a blanket `operator: Exists`. That
one also matches node.kubernetes.io/unreachable:NoExecute with no
tolerationSeconds, which stops admission adding the bounded 300s one and lets
the Pod outlive the Node it runs on.

`proxmox.insecure` defaults to false: the controllers verify the hypervisor's
certificate unless an operator turns that off, and both run here, so the value
describes what the management cluster trusts, not the tenant. The README gains
a section on what the two tokens can reach — the controller acts on whatever
VMID and volume handle the tenant's objects name — and on the per-tenant
Proxmox scope that contains it: a storage of the tenant's own and one ACL on
the resource pool proxmox.pool puts its workers in, which follows every VM
capmox creates — VM.Audit there for the CCM too, which deletes a NotReady Node
whose VM it cannot see.

ccm, csi and insecure are optional in the schema. Each has a default, and what
the substrate needs is enforced at render time — a proxmox cluster without
ccm.credentialsSecretName does not render — so a Kubernetes object written
before these fields existed stays valid, and adding them is not a breaking
change to the API.

The kubelet flag is asserted on both substrates: present on proxmox, absent on
kubevirt, where nothing would ever clear the taint it causes.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-29 21:04:05 +02:00
Marian Koreniuk 93c89c8bf6 feat(kubernetes): add the Proxmox CSI driver, with its controller in this cluster
Tenant clusters on the proxmox substrate get persistent volumes backed by the
hypervisor's storage. The driver is sergelogvinov/proxmox-csi-plugin, vendored
as a chart that reproduces byte-for-byte from `helm pull`.

It is deployed in halves, and which half goes where is the whole design. The
controller authenticates to Proxmox, so it runs HERE, in the tenant's namespace
of the management cluster, and drives the tenant over its admin kubeconfig —
the same shape templates/csi/deploy.yaml already uses for kubevirt-csi. The
tenant gets the node plugin, the CSIDriver object and the storage classes; its
DaemonSet mounts no credentials.

Putting the controller in the tenant would hand the token to the tenant. The
resource definition lists the admin kubeconfig in secrets.include, so the
tenant is cluster-admin of its own cluster and can read any Secret there, and
the chart writes the driver's config.yaml — token included — wherever it is
installed. The privileges the driver documents are granted on `/`, so that
token reaches every VM and every volume on the Proxmox cluster, not just this
tenant's.

Three consequences of the split are handled here rather than discovered later:

  - The vendored chart has no switch for its controller, so the tenant release
    sets replicaCount: 0. The Deployment object still renders; no Pod does.
  - Its Secret template is gated on a non-empty config.clusters, so an empty
    list is what keeps the credential out.
  - CSIStorageCapacity is published by the provisioner sidecar, which now runs
    here and owns those objects through a Pod that does not exist in the
    tenant. Left on, CSIDriver.storageCapacity=true and the scheduler refuses
    every Pod with "node(s) did not have enough free storage". The chart
    hardcodes that field, so a kustomize postRenderer patches it rather than
    forking the chart.

The controller follows proxmox.csi.enabled, like the tenant release: with the
driver switched off there is nothing for it to serve and no token to hold.

Volumes are owned per tenant. The driver names them vm-<controllerVmID>-pvc-…
and defaults the id to 9999; on shared storage that made every tenant's volumes
indistinguishable to an operator and to a VM-level ACL. Each cluster derives its
own from its release name.

What this does not do is make one tenant unable to reach another's volumes: a
tenant can still create a static PV naming a foreign volumeHandle. That needs a
per-tenant Proxmox scope — a dedicated storage and a VM ACL per tenant — which
is a deployment requirement rather than a chart change.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-29 21:04:05 +02:00
Aleksei Sviridkin 53829ea109 fix(kubernetes): reconcile KCCM EndpointSlices when tenant node VMIs change (#3500)
## What this PR does

Adds `patches/vmi-informer.diff` to the `kubevirt-cloud-provider` image
of the `kubernetes` package. The file is named after what it does: no
upstream pull request carries this change yet
(kubevirt/cloud-provider-kubevirt#382 covered the same desync and was
closed as rotten), so a number would point at an unrelated upstream PR
and invite someone bumping the pin to drop it.

The `kubevirteps` controller derives the infra EndpointSlices of a
LoadBalancer Service from the VMIs backing the tenant nodes
(`getDesiredEndpoints`, and the all-VMIs fallback added by `379.diff`).
But it registers informers only on infra Services, tenant EndpointSlices
and infra EndpointSlices, and both factories are built with a resync
period of `0`. VMI addresses are resolved with a one-off `Get` during a
reconcile.

Consequently a VMI appearing, disappearing or changing its address
triggers no reconciliation, and no periodic relist ever catches up. The
EndpointSlices keep pointing at VMIs that no longer exist and the
LoadBalancer blackholes the share of the inbound connections hashed onto
those dead addresses — sporadic inbound timeouts, with nothing wrong on
the tenant side.

Services whose tenant EndpointSlice churns along with its pods recover
on their own, because that churn produces an event. Cilium Gateway API
Services do not: their tenant slice is a static sentinel
(`192.192.192.192`, no `NodeName`), so nothing ever forces a recompute.
Seen in production on a cluster whose GPU MachineDeployment is rolled by
the cluster-autoscaler: 5 of the 9 endpoints of each of the 3 gateways
pointed at destroyed VMs (13 failures out of 20 requests). Worth noting
for anyone debugging this: the KCCM looks dead (no log for hours) while
its lease is still being renewed every second. The process is healthy,
it is simply never woken up.

The patch:

- adds an informer on the infra `VirtualMachineInstances` that requeues
the LoadBalancer Services of this cluster (those labelled
`cluster.x-k8s.io/cluster-name=<clusterName>`, the ones `reconcile` does
not skip) on add/delete, and on updates filtered by a fingerprint of
what actually shapes an endpoint: the VMI's cluster label,
`Status.NodeName`, `Status.Phase`, and the address of the `default`
interface;
- gives both informer factories a 5 minute resync, as a safety net if an
event is ever missed;
- scopes the all-VMIs fallback to the VMIs labelled
`cluster.x-k8s.io/cluster-name=<clusterName>`. An infra namespace can
hold several tenant clusters as well as standalone `VirtualMachines`,
and their VMIs do not serve this Service's NodePort, so selecting them
adds endpoints that blackhole traffic. The fallback widens to every VMI
only when none of them carries the label, so an environment that does
not label them does not lose all of its endpoints; once any VMI is
labelled, an empty selection means this cluster has no node right now
and is not widened to another cluster's VMIs.

No RBAC change: the KCCM role already grants `watch` on
`virtualmachineinstances`.

On `main`, the Gateway scenario reaches the all-VMIs fallback without
any other KCCM patch: Cilium 1.19.5 (shipped since v1.6.0) sets
`kubernetes.io/service-name` on the tenant slice of a `cilium-gateway-*`
Service (cilium/cilium#46242), so the existing lookup finds the sentinel
slice and falls back on its missing `NodeName`. This is why #3374, which
added that lookup to the KCCM, was closed. On v1.5.x and earlier, which
ship Cilium 1.19.3, a Gateway Service finds no tenant slice at all, so
it has no endpoints to go stale in the first place.

## Testing

- `341 + 379 + vmi-informer` applies onto the pinned `a0acf33`; the
Dockerfile steps (`go get`, `go mod tidy`, `go mod vendor`,
`-mod=vendor` build of the manager) pass, `go vet` and `gofmt` are
clean.
- The patch adds 6 specs, and the suite runs 21 of 21 green: fallback
scoped to the cluster, fallback preserved when no VMI is labelled, no
widening when only another cluster's VMIs are present, unlabelled VMIs
ignored once some are labelled, VMI add/delete requeues the Service of
this cluster and not the one of another cluster, and VMI updates driven
through the fake dynamic client: an annotation-only update does not
requeue, while a change of phase, node, address, or cluster label does.
- Mutation-tested, each one turns the suite red: always requeuing on
update; dropping the phase, the node, the address or the cluster label
from the fingerprint; requeuing every Service of the namespace; dropping
the delete requeue.
- Under `-race`, the suite reports an intermittent data race at
`kubevirteps_controller_test.go:820` (a `PrependReactor` racing an
informer list). It reproduces identically on `341 + 379` without this
patch, and the new specs pass on every `-race` run.
- Validated on a live cluster with an image built from an earlier
revision of this patch (same informer and fallback, before the
cluster-scoped requeue and the label in the fingerprint): endpoint added
in under 2s when a VMI appears, dead address removed in 33s when it is
deleted. Before the patch the dead address stayed forever.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff is one patch file of the KCCM image: it touches no values,
schema, CRD, `hack/` tooling, namespace, telemetry metric or node
prerequisite, so the trigger map comes out empty.

### Release note

```release-note
fix(kubernetes): the KCCM now reconciles the LoadBalancer EndpointSlices when the VMIs backing the tenant nodes change, instead of keeping stale addresses that blackhole part of the inbound traffic
```
2026-09-29 19:32:42 +03:00
Aleksei Sviridkin 924d28cf3d feat(kubernetes): default gpu-operator to an OS-provided driver on Talos (#3585)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

Closes the gpu-operator half of #3563 — gap 3 (the operator cannot
validate a driver it did not install) and gap 4 (nothing routes CUDA
libraries to a tenant pod).

Tenant clusters have run Talos workers since 1.6.0, where the NVIDIA
driver comes from a system extension baked into the worker image rather
than from the operator's driver container. The addon default still
described the Ubuntu shape, so a GPU node group came up with the
operator fighting the OS for the driver and no path at all for the CUDA
libraries.

Three changes, each closing one silent failure:

| Change | Without it |
|---|---|
| `driver.enabled: false`, `toolkit.enabled: false` | the driver
container loads its own module and clashes with the extension's; the
toolkit container rewrites `/etc/containerd/config.toml`, which Talos
owns and which already registers the `nvidia` runtime with `enable_cdi =
true` and `cdi_spec_dirs = ['/run/cdi']` |
| `validator.driver.env`: `DRIVER_ROOT=/`, `NVIDIA_DRIVER_ROOT=/`,
`DISABLE_DEV_CHAR_SYMLINK_CREATION=true` | the driver validator looks
for a driver-container install, finds none, never writes its sentinel,
and the device plugin stays blocked behind `toolkit-validation` — the
loop on `Attempting to validate a driver container installation` (**gap
3**) |
| `devicePlugin.env`: `DEVICE_LIST_STRATEGY=cdi-annotations`,
`NVIDIA_CTK_PATH=/usr/local/bin/nvidia-ctk`, plus `DEVICE_ID_STRATEGY`,
`NVIDIA_DRIVER_ROOT`, `CDI_ANNOTATION_PREFIX` | tenant pods carry no
`runtimeClassName` and did not need one on Ubuntu, where the toolkit
made `nvidia` the default containerd runtime. Talos does not, so CDI is
the only route: the pod gets the device and no CUDA libraries (**gap
4**) |

Every value is the configuration validated on a production Talos GPU
node group, minus the parts that only mattered while the driver
container was in play. No DaemonSet, no new image, no new package — the
fix is values.

### The removed default

The previous default fed `NVreg_NvLinkDisable=1` to the driver container
through the `nvidia-kernel-module-params` ConfigMap. With that container
off the ConfigMap is mounted by nothing, so this removes the default. On
Talos the parameter belongs to the worker machine config, which the
`kernelModules` field of the worker pool writes: that field is top-level
in the KubernetesNodes app (`packages/apps/kubernetes-nodes`), added by
#3571, and its automatic set for a pool holding an `nvidia.com/*` GPU
already loads `nvidia` with `NVreg_NvLinkDisable=1`. The wrapper chart
keeps the ConfigMap feature for an operator who re-enables the driver
container through `valuesOverride`. A test pins the absence so the dead
default cannot come back and offer a second, inert source for the same
setting.

### Relationship to #3571

Independent files, no conflict, but only useful together: this makes
`driver.enabled: false` correct, and #3571 is what actually loads the
modules the extension ships. Merged alone, this changes a GPU node group
from "operator fights the OS" to "operator stays out of the way and the
modules still are not loaded" — better, but not yet working. #3571 alone
leaves the operator fighting for the driver. Either order is safe; both
are needed for a GPU node group to work.

### What is verified, and what is not

Verified here: the rendered HelmRelease values, that `valuesOverride`
still wins over each new key, and that the removed default is gone. The
individual values are the ones running in production today, applied by
hand through `addons.gpuOperator.valuesOverride`.

Not verified here: this exact default exercised end-to-end as the
chart's own output. That needs a Talos GPU node group whose modules
load, which needs #3571 released. I would rather say so than imply a
green e2e that does not exist — happy to hold this as draft until I have
run a single-node L40S cluster against the chart default, and I will
report the result on this PR either way.

### Not covered

HAMi. When `addons.hami.enabled` the operator's device plugin stays off
and HAMi serves the GPUs; HAMi's own plugin needs equivalent CDI wiring
on Talos, which this default does not attempt. Flagging rather than
guessing — the production cluster this was derived from runs HAMi
disabled, so I have no validated configuration for that path. Worth its
own issue if a maintainer confirms the shape.

Also not in scope, and unchanged from the merge base:
`talos.installerRepository` and `talos.version` reach the worker machine
config unvalidated (being tracked separately by @lexfrei), and the
`nvidia-operator-validator` chain for the *management* cluster's own
passthrough/vgpu variants, which never load the host driver at all.

### Screenshots

No UI changes.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/643
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(kubernetes): The GPU Operator addon of tenant Kubernetes clusters now defaults to a driver provided by the operating system, which is the shape of the Talos workers these clusters have run since 1.6.0. `driver` and `toolkit` are disabled (the driver comes from a Talos system extension, and containerd configuration belongs to the machine config), the driver validator is pointed at the host root so it stops looping on "Attempting to validate a driver container installation", and the device plugin is switched to CDI (`DEVICE_LIST_STRATEGY=cdi-annotations`) so tenant pods receive the CUDA libraries — previously they received the GPU device and nothing else, since Talos does not make `nvidia` the default containerd runtime. The previous default, which fed `NVreg_NvLinkDisable=1` to the operator's driver container through a ConfigMap, is removed because that container is disabled; on Talos the parameter is carried by the worker machine config, through the `kernelModules` field of the worker pool in the KubernetesNodes app (`packages/apps/kubernetes-nodes`), which loads `nvidia` with `NVreg_NvLinkDisable=1` by default on a pool holding an `nvidia.com/*` GPU. Operators who overrode `addons.gpuOperator.valuesOverride` to work around this by hand can drop those overrides. NOTE: an override that sets `devicePlugin.env` replaces the list rather than merging it, so restate the full list when overriding it or CDI stops working. HAMi is not covered: when `addons.hami.enabled` the operator's device plugin stays off and HAMi's own plugin needs equivalent CDI wiring, which this change does not attempt.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Configuration**
* GPU workloads on Talos workers now use host-provided NVIDIA drivers
and tooling by default; the GPU Operator’s driver and toolkit are
disabled.
* Device discovery uses CDI annotations and UUID-based device IDs. When
HAMi is enabled, the Operator’s device plugin is disabled.
  * Host-root validation no longer creates `/dev/char` symlinks.
  * Driver kernel-module configuration is no longer created by default.

* **Documentation**
* Clarified when kernel-module configuration applies and what settings
are needed to re-enable the driver.
  * Noted that `valuesOverride` replaces lists wholesale.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 19:28:17 +03:00
Mattia Eleuteri 1f9b1b85db fix(kubernetes): reconcile KCCM EndpointSlices when tenant node VMIs change
The kubevirteps controller derives the infra EndpointSlices of a
LoadBalancer Service from the VMIs backing the tenant nodes, but it
watches no VMI and its informers had no resync period. A VMI appearing,
disappearing or changing its address therefore triggered no
reconciliation, and no periodic relist ever caught up: the EndpointSlices
kept pointing at VMIs that no longer existed and the LoadBalancer
blackholed the share of the inbound connections hashed onto those dead
addresses.

Services backed by a tenant EndpointSlice that churns with its pods
recovered on their own. Cilium Gateway API Services did not: their
tenant slice is a static sentinel, so nothing ever forced a recompute.
Observed in production on a cluster whose GPU MachineDeployment is
rolled by the cluster-autoscaler: 5 of the 9 endpoints of every gateway
pointed at destroyed VMs.

Add an informer on the infra VirtualMachineInstances that requeues the
LoadBalancer Services of this cluster, ignoring the updates that cannot
change an endpoint, and give the informers a 5 minute resync as a
safety net. Services of other tenant clusters sharing the namespace are
not requeued, since reconcile would skip them.

Also scope the all-VMIs fallback to the VMIs labelled with the name of
the cluster being reconciled: an infra namespace can hold several
tenant clusters and standalone VirtualMachines, whose VMIs do not serve
the Service NodePort. The fallback widens to every VMI only when none of
them carries the label, so that environments that do not label them keep
their endpoints. Once any VMI is labelled, an empty selection means this
cluster has no node right now, and widening it would publish the
addresses of another cluster.

The patch is named after what it does rather than after an upstream
pull request, because none carries this change yet.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
2026-09-29 17:21:19 +02:00
Mattia Eleuteri 669fb2416a feat(kubernetes): default gpu-operator to an OS-provided driver on Talos
Closes the gpu-operator half of #3563. Tenant clusters have run Talos
workers since 1.6.0, where the NVIDIA driver comes from a system extension
baked into the worker image rather than from the operator's driver
container. The addon default still described the Ubuntu shape, so a GPU
node group came up with the operator fighting the OS for the driver and no
path at all for CUDA libraries to reach a tenant pod.

Three things change, each fixing one silent failure:

driver.enabled and toolkit.enabled go false. The driver container would
load its own module and clash with the extension's; the toolkit container
would rewrite /etc/containerd/config.toml, which Talos owns and which
already registers the nvidia runtime with enable_cdi = true and
cdi_spec_dirs = ['/run/cdi'].

validator.driver.env points DRIVER_ROOT and NVIDIA_DRIVER_ROOT at / and
disables /dev/char symlink creation. The driver validator otherwise looks
for a driver container install, finds none, never writes its sentinel, and
leaves the device plugin blocked behind toolkit-validation — the loop on
"Attempting to validate a driver container installation".

devicePlugin.env selects cdi-annotations and points NVIDIA_CTK_PATH at the
host's nvidia-ctk. Tenant pods carry no runtimeClassName and did not need
one on Ubuntu, where the toolkit made nvidia the default containerd
runtime; Talos does not, so CDI is the only route for the CUDA libraries.
DEVICE_ID_STRATEGY, NVIDIA_DRIVER_ROOT and CDI_ANNOTATION_PREFIX are set
and tested alongside it: CDI device names key on the UUID strategy, the
plugin resolves the driver from NVIDIA_DRIVER_ROOT, and the annotation
prefix has to be the one containerd reads, so a regression in any of them
breaks device resolution on its own.

The previous default fed NVreg_NvLinkDisable=1 to the driver container
through the nvidia-kernel-module-params ConfigMap. With that container off
the ConfigMap is mounted by nothing, so the default is removed. On Talos a
module parameter belongs to the worker machine config, which the
kernelModules field of the KubernetesNodes app
(packages/apps/kubernetes-nodes, #3571) writes. The wrapper chart keeps
the ConfigMap for an operator who re-enables the driver container.

Every value here is the configuration validated on a production Talos GPU
node group, minus the parts that only mattered while the driver container
was in play.

Not covered: HAMi. When addons.hami.enabled the operator's device plugin
stays off and HAMi serves the GPUs, and HAMi's own plugin needs equivalent
CDI wiring on Talos that this default does not attempt.

Refs: #3563

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:09:13 +02:00
Aleksei Sviridkin 2b7e0db40f fix(kubernetes): let kubevirt-csi hot-unplug from an orphaned VMI
The driver falls back to the VMI-level removevolume subresource when
a volume is still attached to the VMI but no longer in the VM spec.
The -kcsi Role grants only the VM-level subresources, and nothing
else bound to the service account covers the VMI one: kubevirt.io:
default grants no subresource of it, and the kubevirt.io:edit and
admin roles that do are aggregated only into the built-in edit and
admin roles, which the tenant namespace never binds to this account.
Every such call ends in 403, the unpublish retries until it fails,
and the hot-plug pod keeps the infra volume attached.

The call stays needed once the driver stops falling back for a
VM-owned VMI (kubevirt/csi-driver#382), because it still makes it
for a VMI no VM owns. Deleting a VM with orphan propagation leaves
its VMI running, since the KubeVirt VM controller then drops its
finalizer without stopping the VMI, so the tenant node stays up and
its volumes stay attached until the driver detaches them.

The grant adds no reach the account lacks: it can already remove a
volume from any VM in the namespace, and virt-api refuses the VMI
call on a VM-owned VMI once the HotplugVolumes gate is off.
virtualmachineinstances/addvolume stays ungranted, as the driver
only attaches through the VM.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 12:28:31 +03:00
Aleksei Sviridkin 3e91da913e fix(kubernetes): log kubevirt-csi temp NFS mount dir removal failure
The NFS publish path temp-mounts the export root to create /data and
migrate old files, then removes the temp dir in a defer that dropped
the error. When the unmount before it fails, the removal fails on a
busy mount point and the leftover dir under the node's temp dir went
unreported.

The failure is logged rather than returned: a leftover temp dir does
not make the volume unusable, and failing an otherwise good publish
would only make kubelet retry it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 12:23:39 +03:00