Commit Graph
181 Commits
Author SHA1 Message Date
ZverGuy 95dc591198 test(kubernetes): keep application identity out of guest log streams
Guest pods are tenant-controlled, so their lineage labels are not a
trustworthy application identity, and the platform must not promote
them to the stream fields the parent scopes application logs by. The
guest release replaces the monitoring-agents filters and outputs
wholesale, which is what keeps the parent's label lift out of guests;
pin that so a later change to the guest overrides cannot bring it in.

This only covers what the platform ships. A guest admin can still edit
the guest's fluent-bit or write to the parent's vlinsert directly; that
needs a write-path control, which the logs proposal leaves open.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-04 04:18:09 +07:00
Aleksei Sviridkin ae9e7c4c64 test(kubernetes): pin RetryOnFailure on the Proxmox csi release (#4969)
The KubeVirt csi and cilium releases carry a helm-unittest pin on
`strategy.name: RetryOnFailure` for both actions, and the e2e
HelmRelease guard in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` names
those two tests as load-bearing. `helmreleases/csi-proxmox.yaml` sets
the same strategy for the same reason and had no pin.

This adds
`packages/apps/kubernetes/tests/csi_proxmox_install_retry_test.yaml`, a
mirror of `csi_install_retry_test.yaml` rendered on the proxmox
substrate, and updates the guard comment that pointed at the gap. No
template change.

Verified with `helm unittest .` in `packages/apps/kubernetes` (32
suites, 349 tests).

Closes #4731


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
* Expanded automated coverage for Proxmox CSI installation and upgrade
retry behavior, including the retry interval and remediation retry
limits. The checks verify that retries continue according to the
configured settings during installation and upgrades. This update does
not change user-facing functionality.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-28 18:30:06 +03:00
Andrei Kvapil b43196e549 fix(kubernetes): pin the Proxmox cloud-config builders to the bundled image
The Proxmox substrate added two init containers that build the cloud
config from the hypervisor token Secret, and both still took their
image from images.waitForKubeconfig. With that key gone the render
failed on a nil map; with it set, a tenant chose the image that reads
the token. They now run the bundled busybox through cozy-lib.image, like
the wait-for-kubeconfig container next to them.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:17 +05:00
Andrei Kvapil 1636349a63 fix(api): point the removed image overrides at node registry mirrors
The warning for a removed image override told an air-gapped install to
use the platform-wide registry, but nothing in the platform writes
_cluster.images-registry, so there is nothing to set. What serves these
images offline is what serves every other platform image: the registry
mirrors configured on the nodes, as the air-gapped install guide sets
up. The tests that hold the cozy-lib.image routing now say what they
check rather than promising a platform knob.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:16 +05:00
Andrei Kvapil dfff5c2dac test(kubernetes): hold the routing of the busybox and unpin images
The busybox helper shared by four workloads and the pool's pre-delete
Job had no test holding their route through cozy-lib.image, so
reverting either line left both suites green.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:16 +05:00
Andrei Kvapil d99a1b9f2f fix(kubernetes): route the helper images through cozy-lib.image
These images now go through cozy-lib.image like the harbor, kafka and
mariadb hook images, so they follow _cluster.images-registry wherever
that is set rather than any per-application value. An air-gapped
install serves them the way it serves every other platform image,
through the registry mirrors configured on its nodes.

The talos-csr-signer sidecar and the OIDC Jobs had no test holding
their image, which is where an override could come back unnoticed.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:15 +05:00
Andrei Kvapil af20694359 fix(kubernetes)!: drop the helper image overrides
images.waitForKubeconfig, images.kubectl and images.talosCsrSigner
ran whatever image a tenant named inside the management cluster: the
csr signer as a sidecar of the tenant control plane with the Talos CA
key mounted, the other two in pods holding service account tokens for
CAPI, KubeVirt and Cilium objects in the tenant namespace. A tenant
never gets that access through the apps API, and the platform should
not hand it over through an image field.

The images now always come from the bundled tag files. A release that
still carries the keys renders exactly as one without them.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:14 +05:00
Marian Koreniuk 98d0402096 test(kubernetes): pin RetryOnFailure on the Proxmox csi release
The KubeVirt csi and cilium releases carry a helm-unittest pin on
strategy.name: RetryOnFailure for both actions, and the e2e HelmRelease
guard names those two tests as load-bearing. helmreleases/csi-proxmox.yaml
sets the same strategy for the same reason and had no pin. Add the mirror
test and update the guard comment that pointed at the gap.

No template change.

Closes #4731

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-10-19 23:08:02 +02:00
Ray Low 904bc38fc3 fix(kubernetes): delete the Talos PKI and oidc-bootstrap Job on uninstall
cert-manager runs with enableCertificateOwnerRef: false, so the
<release>-talos-ca and <release>-talos-tls-cert Secrets carry no
ownerReference to their Certificates and outlive the release, keeping
the CA private key of a cluster that no longer exists. A new cluster
created under the same name then adopts that stale CA instead of
issuing its own trust root.

The <release>-oidc-bootstrap Job is a hook whose only delete policy is
before-hook-creation. Hooks are not part of the release manifest, so
uninstall never removes it.

The pre-delete hook deletes the two Certificates before their Secrets,
because deleting a Secret while its Certificate exists makes
cert-manager re-issue it. It runs after the TenantControlPlane is gone,
since the control plane's CSR signer mounts the CA. The Role grants
each delete by resourceName only.

Assisted-by: LLM
Signed-off-by: Ray Low <crossrt@gmail.com>
2026-10-11 22:40:47 +08:00
Aleksei Sviridkin 5486d4ffcf fix(networking): harden webhook policy assumptions (#4058)
## What this PR does

This is follow-up for
[#3997](https://github.com/cozystack/cozystack/pull/3997). It reads
ingress-nginx admission port from effective subchart values and refuses
empty value, and fixes `kubeProxyReplacement=false` status because
cozystack has no supported Service translator in that state. Gateway
test now requires one real ClusterIP and keeps same ready Pod UID, IP
and port for whole probe, so stale EndpointSlice, replaced Pod or wrong
Service can't report deny. Policy selectors and two-ended deny from
[#3997](https://github.com/cozystack/cozystack/pull/3997) stay
unchanged, and this does not claim root cause for
[#3983](https://github.com/cozystack/cozystack/issues/3983).

### Screenshots

Not applicable.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(networking): use effective ingress admission port, report unsupported Cilium service routing correctly, and reject stale backend evidence in gateway tests
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Admission webhook connectivity checks now validate the configured
Service and its exact backend endpoint throughout probing, improving
failure detection.
- Admission webhook policies now fail closed when the webhook port is
not explicitly configured.

- **Documentation**
- Clarified that supported service routing requires
`kubeProxyReplacement: true`.
- Updated guidance to explain that disabling this setting or webhook
policies does not provide a supported fallback.

- **Tests**
- Added coverage for missing webhook port configuration and strengthened
networking stability checks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 08:37:53 +03:00
Myasnikov Daniil c9558f5658 fix(networking): correct webhook policy assumptions
Read the ingress admission port from the effective subchart values and fail if the port is absent. Describe kube-proxy replacement as a required service-routing invariant while retaining target-side enforcement.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:00:03 +03:00
Aleksei Sviridkin 520af57905 chore: drop private cluster, host and personal names from the tree
Private infrastructure names, a real hostname, a commented-out
internal mirror block, an internal product name, and a reviewer's
name in a comment do not belong in a public repository; drop them
and replace what needs a stand-in with neutral placeholders.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-07 04:38:06 +03:00
Aleksei Sviridkin bcda5c893a feat(kubevirt)!: drop HotplugVolumes for declarative volume hotplug (#4453)
## What this PR does

This turns off the deprecated KubeVirt `HotplugVolumes` gate and moves
the platform to declarative volume hotplug. It changes how every volume
hotplug on the platform works, so the title carries `!`. Closes #4032.

This PR is stacked on #4451 and has to merge after it. #4451 adds the
kubevirt-csi flag this PR flips.

The replacement gate is already on in KubeVirt 1.9, and the old gate is
the only thing keeping it off. `DeclarativeHotplugVolumes` is Beta at
v1.9.0, and Beta gates are on by default. virt-controller skips
`handleDeclarativeVolumeHotplug` while `HotplugVolumes` is on, so
dropping the old gate is what switches the new one on.

The kubevirt-csi fallback goes off in the same change. With the old gate
off, virt-api rejects the VMI-level `removevolume` on a VM-owned VMI, so
the chart now passes `--enable-vmi-hotplug-fallback=false`. The comments
next to the gate and the flag point at each other.

This needs the rebuilt kubevirt-csi image. A driver built before #4451
does not know the flag and exits on it at startup. The image is stamped
at release preparation and ships in the same release as this change, so
only a `make apply` from main before then would crash-loop the CSI
controller.

What changes at runtime on an upgrade:

- The VM-level `addvolume` and `removevolume` calls now edit `VM.spec`
directly. virt-controller then makes the VMI hotplug volumes match it,
instead of going through `VM.status.volumeRequests`.
- Volumes hot-plugged through the VM subresource are in `VM.spec`, so
they stay attached.
- During a tenant teardown, when the VM goes before its VMI, the driver
no longer calls the VMI subresource. It waits for the VMI status to drop
the volume, and the garbage collector deletes the VMI with its VM.
Upstream waits up to two minutes. If the VMI is still there after that,
the unpublish fails and is retried.

A new platform migration keeps old VMI-only volumes attached. On its
first pass virt-controller removes every hotpluggable VMI volume that is
missing from `VM.spec`. Until July 2025 kubevirt-csi attached volumes at
the VMI level (the Role then granted
`virtualmachineinstances/addvolume`), and live migration carries such a
VMI across KubeVirt updates. Without help, a tenant node not recreated
since then would lose a mounted disk on upgrade.

Migration 59 copies each such volume and its disk from the VMI into the
VM template before anything else changes. Adding a hotplug volume to the
VM spec is a valid live update in KubeVirt, so no VM needs a restart. It
runs as the platform chart's pre-upgrade hook, before the upgrade
re-points `cozystack-packages`. The KubeVirt CR and the tenant
kubernetes charts both render from that source, so the gate and the flag
cannot change before the migration has finished. It only touches VMs
that need it, logs each one, and a second run changes nothing.

The migration also makes ephemeral hotplugs permanent. A volume a user
added at the VMI level, for example with `virtctl addvolume` without
`--persist` from an older virtctl, has the same shape as kubevirt-csi's
old attachments, because virt-api builds both the same way. There is
nothing reliable to tell them apart by, so both are copied. Leaving the
user's volume out would not keep it ephemeral: the first reconcile would
unplug it from the running VM. After the upgrade it is part of `VM.spec`
and attaches again on a restart, and `virtctl removevolume` drops it.

Ephemeral hotplug stops working on VMs. With the old gate off, virt-api
refuses VMI-level `addvolume` and `removevolume` on a VMI that a VM owns
(`vmiVolumePatch` in v1.9.0). A volume on such a VMI is added and
removed through the VM from then on. A VMI without a VM is not affected.

Only running VMIs are copied. virt-controller reconciles no other VMI,
and a stopped VM builds its next VMI from `VM.spec`, so copying from a
stopped one would bring back volumes the restart was going to drop.

Migrations run only on the isp-full, isp-full-generic and isp-hosted
variants. KubeVirt comes from the platform only on isp-full and
isp-full-generic, so those are covered. On the default variant, where
packages are installed by hand, nothing runs the migration.

This optional check lists the VMIs and volumes the migration would copy.
It uses the same rule. On the default variant, run it before upgrading
and recreate each tenant node it lists:

```bash
kubectl get vm,vmi --all-namespaces --output json | jq --raw-output '
  ([.items[] | select(.kind == "VirtualMachine")
    | {key: "\(.metadata.namespace)/\(.metadata.name)", value: [.spec.template.spec.volumes[]?.name]}]
    | from_entries) as $vm
  | .items[] | select(.kind == "VirtualMachineInstance")
  | ([.metadata.ownerReferences[]? | select(.controller == true and .kind == "VirtualMachine" and (.apiVersion | startswith("kubevirt.io/"))) | .name][0]) as $owner
  | "\(.metadata.namespace)/\($owner)" as $key
  | select(.status.phase == "Running" and $owner != null and $vm[$key] != null)
  | .spec.volumes[]? | select(.dataVolume.hotpluggable == true or .persistentVolumeClaim.hotpluggable == true)
  | .name as $n | select($vm[$key] | index($n) | not)
  | "\($key) \($n)"'
```

Empty output means nothing is affected. I checked the query against the
migration's test fixtures only. Neither the query nor the migration has
run on a live cluster.

#4287 also adds a migration 59. If it merges first, this one moves to
60: the file, the version it stamps, `migrations.targetVersion`, the
bats file (its `run_migration` argument and expected `STAMP`) and
`hack/testdata/migration-59-vmi-hotplug`.

A VM with a pending hotplug request at upgrade time may also get stuck.
virt-controller still processes `VM.status.volumeRequests` before the
declarative handler, and it applies them through the VMI subresource,
which virt-api rejects once the old gate is off. The sync then fails
before the VM spec is updated, so nothing clears the request, and every
later sync of that VM fails the same way. A retry through the VM
subresource edits the spec and clears it, which covers kubevirt-csi's
own requests. A one-shot request stays stuck until the VM restarts. I
read this from the v1.9.0 code and haven't reproduced it. Waiting until
this lists nothing before upgrading narrows the window, but a request
made during the rollout can still hit it:

```bash
kubectl get vm --all-namespaces --output json | jq --raw-output '.items[] | select((.status.volumeRequests // []) | length > 0) | .metadata.namespace + "/" + .metadata.name'
```

The two halves live in different HelmReleases, so there is a short
window during the rollout where the driver and the gate disagree. In
that window an unpublish fails and is retried until both are updated.

A tenant cluster whose `kubernetes` HelmRelease is suspended or failing
keeps its old driver for longer. Drivers from before July 2025 attach
through the VMI subresource, so on such a tenant every attach is refused
until its release reconciles. Such a driver can also attach a volume
after the migration has run and before the gate flips. The migration
doesn't see that volume, and the first declarative reconcile unplugs it
from the running VM.

The migration lists the VMs once and patches each one with a
`resourceVersion` test, so a VM that changes in between fails the hook
Job. The Job has `backoffLimit: 3`, and each retry lists again and skips
the VMs already patched. Re-reading only the conflicting VM would avoid
the retry; I left that out.

The new gate is listed in the CR even though it is on by default. With
the old gate gone it is the only hotplug path, and the chart refuses a
render that disables it. I found no other user of `HotplugVolumes` in
the platform or in the downstream repositories.

The VMI-level fallback may have been failing already. The kcsi Role has
no rule for the VMI-level `removevolume` (#4452), so that call likely
got 403 with or without the gate. I haven't checked that on a cluster.

`hack/migration-59-vmi-hotplug-volumes.bats` runs the migration against
a fake kubectl: a VM that needs the copy, VMs and VMIs that must be left
alone (a VMI that is not Running among them), a second run, a cluster
without KubeVirt and the fail-closed paths. The helm-unittests for the
KubeVirt CR pin the whole gate list, accept `HotplugVolumes` in
`disabledFeatureGates` and refuse `DeclarativeHotplugVolumes` there. The
CSI test pins the flag at false, and the `make update` idempotency test
still passes. None of that exercises hotplug itself. The real test is
e2e with volume attach and detach on a tenant cluster, including a
teardown. No CI lane upgrades from the previous release yet, so an
upgrade with a live VMI-only volume is untested; #4463 tracks that test.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kubevirt)!: KubeVirt now runs with DeclarativeHotplugVolumes instead of the deprecated HotplugVolumes feature gate, and kubevirt-csi no longer uses the VMI-level hot-unplug. Volume hotplug requests are applied through VM.spec. The upgrade copies volumes that older kubevirt-csi versions attached only to the VMI into VM.spec, so they stay attached; the check in the PR description is optional. This also applies to ephemeral volumes added to running user VMs at the VMI level, such as with virtctl addvolume without --persist: they become part of VM.spec and attach again after a VM restart. virt-api now refuses VMI-level addvolume and removevolume on a VMI owned by a VM, so such volumes are added and removed through the VM. A tenant cluster whose kubernetes release is suspended or failing keeps its old kubevirt-csi, and a driver from before July 2025 cannot attach volumes there until that release reconciles, and a volume it attaches during the upgrade itself can be unplugged again. A values override that puts DeclarativeHotplugVolumes into disabledFeatureGates now fails the kubevirt render. Clusters on the default variant do not run platform migrations and should run that check before upgrading.
```





<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
  * Declarative hotplug volumes are enabled by default for KubeVirt.
* Running virtual machines keep eligible hotplugged volumes and disks
when their configuration is migrated.
* **Bug Fixes**
* The migration avoids adding duplicate volumes or disks and does not
mark the migration complete if required operations fail.
  * The Kubernetes CSI driver no longer uses the VMI hotplug fallback.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 17:06:03 +03:00
IvanHunters 86bf2ba8ec fix(kubernetes): let kubevirt-csi hot-unplug from an orphaned VMI (#4563)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

This lets the kubevirt-csi controller detach a volume from a VM instance
that no VM owns. Right now that call is always denied. Closes #4452.

The driver has a fallback for a volume that is still attached to the VMI
but is no longer in the VM spec. It calls the VMI-level `removevolume`
subresource. The `-kcsi` Role grants only the two VM-level subresources,
so the call ends in 403. Nothing else reaches this service account with
that grant. Only its own Role and ClusterRole name it as a subject, and
the group bindings that cover every service account grant nothing on
this subresource. At v1.9.0, `kubevirt.io:default` grants only
`kubevirts`, `version` and `guestfs`. `kubevirt.io:edit` and
`kubevirt.io:admin` do include the VMI subresource, but they aggregate
only into the built-in `edit` and `admin` roles, and no tenant binding
gives those to this account. The cozystack tenant roles aggregate on
their own labels and don't list it.

The fallback is still needed after the flag in #4453 goes to false. The
flag only skips VMIs that a VM owns. When a VM is deleted with
`propagationPolicy=Orphan`, the KubeVirt VM controller drops its
finalizer and leaves the VMI running (`vm.go` at v1.9.0). The tenant
node stays up with its volumes attached. When a pod moves off that node,
the driver has to detach through the VMI, and on a 403 the hot-plug pod
keeps holding the infra volume. Until #4453 lands, the fallback also
runs for VM-owned VMIs and gets the same 403.

The grant is `update` on `virtualmachineinstances/removevolume` and
nothing more. The account can already remove a volume from any VM in the
namespace, so it gets no new reach. virt-api also rejects the VMI-level
call on a VM-owned VMI once `HotplugVolumes` is off.
`virtualmachineinstances/addvolume` stays out because the driver only
attaches through the VM.

The new helm-unittest pins the Role's `subresources.kubevirt.io` rule to
these three subresources with `update`. I haven't reproduced the 403 on
a cluster. This shows it on a tenant namespace, before and after:

```bash
kubectl auth can-i update virtualmachineinstances.subresources.kubevirt.io --subresource=removevolume \
  --namespace <tenant-ns> --as system:serviceaccount:<tenant-ns>:<release>-kcsi
```

Upstream kubevirt/csi-driver has the same gap in
`deploy/infra-cluster-service-account.yaml`.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubernetes): the kubevirt-csi controller can now hot-unplug a volume from a VMI that no VM owns, instead of failing with 403 Forbidden.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved Kubernetes CSI volume removal for virtual machine instances,
including cases where a volume is not listed in the virtual machine
specification. This helps the driver complete volume removal in
scenarios where the Kubernetes virtualization API accepts the
instance-level request. Existing permissions for adding and removing
volumes from virtual machines remain unchanged.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 15:14:19 +03:00
IvanHunters 94abf6940b feat(kubernetes): let kubevirt-csi detach without the HotplugVolumes gate (#4451)
## What this PR does

This lets kubevirt-csi detach volumes without the deprecated KubeVirt
`HotplugVolumes` gate. The default stays as it is, so nothing changes at
runtime yet. Part of #4032.

The driver got a switch upstream for exactly this, and I bumped to it.
kubevirt/csi-driver#382 added `--enable-vmi-hotplug-fallback`, merged as
cb4770e4e475. With it set to false, the driver no longer calls the
VMI-level `removevolume` on a VMI owned by a VM. virt-api rejects that
call once `HotplugVolumes` is off.

Our wrapper now passes the flag to upstream and no longer has its own
hot-unplug path. `unpublishHotplugVolume` was a backport of
kubevirt/csi-driver#184, and the previous pin already contained that
change. Upstream detaches from the VMI when the VM is gone, then waits
for the VMI status to drop the volume before it returns. So the
wrapper's re-check never found the volume attached. Adding the ownership
check there would have been dead code.

Closes #4450

The chart leaves the flag at its default of true and does not render it.
The driver image in the chart is stamped only at release preparation, so
between releases main runs the previous release's binary, which does not
know the flag. Go's `flag` package exits on an unknown flag. Rendering
`=true` would crash-loop the CSI controller for no effect. The right
value depends on the KubeVirt CR, not on the tenant, so the comment next
to `HotplugVolumes` in `kubevirt-cr.yaml` points at the controller
Deployment. The follow-up PR drops the gate and renders the flag as
false.

Nothing changes on the node side. Main already pins f5de10d39556, and
cb4770e4e475 is that commit plus kubevirt/csi-driver#382.

A Go unit test runs an unpublish through the wrapper during a teardown,
with the flag on, off with a VM-owned VMI, and off with an orphaned VMI.
It checks that the flag reaches upstream and that only upstream makes
the VMI-level call. Upstream's own tests cover the ownership rule. A
second test holds that an NFS volume only releases the node from its
CiliumNetworkPolicy and never reaches the upstream hot-unplug. A
helm-unittest pins that the chart renders no such argument. I haven't
built the image or run e2e. Volume attach and detach on a tenant cluster
in e2e is the real check.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kubernetes): kubevirt-csi supports --enable-vmi-hotplug-fallback, so it can detach volumes without the KubeVirt HotplugVolumes feature gate. The flag stays true for now and behaviour is unchanged.
```
2026-10-05 10:46:12 +03:00
Aleksei Sviridkin 64f43f4f77 fix(computeplane): do not grant tenants the cluster admin kubeconfig
The computeplane module deploys the kubernetes chart into the tenant
namespace, and that chart's dashboard Role lets the tenant's "use"
subjects read <release>-admin-kubeconfig. Withholding the Secret from
the tenant secrets view does not help when RBAC grants it directly, so
any tenant user at "use" level or above could fetch the cluster-admin
credential of a cluster the module promises to keep operator-owned.

The kubernetes chart now drops that rule when the internal value
withholdAdminKubeconfig is set, and the module sets it in the cluster
HelmRelease's inline values. The tenant cannot reach those values: the
module reads only nodeGroups from them and splices an allowlisted
subset into the pool releases, and Flux applies spec.values over
valuesFrom. A regular kubernetes app keeps the rule, so its dashboard
can still offer the kubeconfig.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:36:57 +03:00
Aleksei Sviridkin fccb7fee06 test(tests): fail absence-only cases when the template renders nothing
Each of these templates is wrapped in a top-level condition, and the
case asserts only that a field is absent. If a values change turns the
condition off, the render is empty and the absence holds trivially, so
the case passes without looking at anything. A document selector makes
an empty render fail with "document not found".

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 22:27:17 +03:00
Aleksei Sviridkin d6b38f0a79 fix(kubernetes): fail the OIDC bootstrap Job when the tenant CA is missing (#4232)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

In `spec.oidc.mode: System`, the post-install/post-upgrade bootstrap Job
patches the chart-owned `<release>-oidc-kubeconfig` Secret with a real
kubeconfig built from the tenant admin kubeconfig's CA data. When that
CA data was empty, the script logged a WARNING and continued, then
exited 0. The hook was satisfied, the HelmRelease reported `Ready`, and
the Secret was left holding the install-time placeholder kubeconfig,
which cannot authenticate. This PR makes the Job fail (`exit 1`)
instead, so the failure surfaces as a red install and can be retried
(the Job already has `backoffLimit: 6` and `restartPolicy: OnFailure`)
rather than reported as a falsely healthy release. This is the smallest
of the three directions raised on the linked issue, and matches how the
neighbouring failure mode in the same script already behaves (a failed
`kubectl apply` of the ClusterRoleBindings already fails the Job under
`set -euo pipefail`). Behaviour for `mode: None` and `mode:
CustomConfig` is unchanged, and the common case — CA data present — runs
the identical patch commands as before.

A regression test was added to
`packages/apps/kubernetes/tests/oidc_test.yaml` asserting the rendered
Job script contains the `exit 1` on empty `CA_DATA` and no longer
contains the old warn-and-continue text. Confirmed it fails against the
pre-fix template and passes against the fix.

Validation:
- `helm unittest .` in `packages/apps/kubernetes` — 150/150 tests pass
(up from 149 before the new test).
- `helm template ... --show-only templates/oidc-rbac-job.yaml` with
representative `System`-mode values renders valid YAML; the extracted
bootstrap script passes `bash -n` and `shellcheck -s sh` introduces no
new warnings versus the pre-change script.
- `make generate` was not run/needed — this change touches neither
`values.yaml`, `values.schema.json`, `Chart.yaml`, nor `README.md`.
- Not run: a live cluster reproduction (Kamaji-issued admin kubeconfig
actually missing CA data at Job run time). The defect is provable from
the script logic itself and the added unit test demonstrates the
fail-then-pass behaviour; there is no live cluster available in this
environment.

### Screenshots

Not applicable — no UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

This change is confined to the shell logic inside
`packages/apps/kubernetes/templates/oidc-rbac-job.yaml`'s post-install
hook Job (a `Values.oidc.mode: System`-only failure path) and its
accompanying helm-unittest test. It does not add, rename or remove a
package, a `values.yaml` key, a CRD, a namespace, a release-owned object
name, or any other surface the trigger map lists, so no downstream
repository is affected.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubernetes): a System-mode OIDC install no longer reports Ready while the tenant OIDC kubeconfig Secret still holds the install-time placeholder — the bootstrap Job now fails (and retries) instead of only logging a warning when the tenant cluster's CA data isn't available yet.
```

Fixes #3765


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* OIDC setup now fails with an error if the tenant admin kubeconfig is
missing certificate authority data, rather than leaving the OIDC
kubeconfig Secret at its install-time placeholder.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-30 01:50:22 +03:00
Pujitha Paladugu 4fe45eba32 fix(kubernetes): fail the OIDC bootstrap Job when the tenant CA is missing
In mode=System the post-install hook Job patches the chart-owned
<release>-oidc-kubeconfig Secret with a real kubeconfig built from the
tenant admin kubeconfig's CA data. When that CA data was empty the
script only warned and exited 0, so the hook was satisfied, the
HelmRelease reported Ready, and the Secret kept the install-time
placeholder kubeconfig, which cannot authenticate. The failure only
showed up when a user tried to use the downloaded kubeconfig.

Exit 1 instead, so the install fails visibly and is retried. The Job
already has backoffLimit: 6 and restartPolicy: OnFailure, and a failed
ClusterRoleBinding apply in the same script already fails the Job the
same way.

Assisted-by: LLM
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
2026-09-30 00:36:44 +03:00
Marian Koreniuk 36b04218d7 test(kubernetes): assert where the hypervisor token is and where it is not
Every assert here was written against the mutation it exists for, and each was
checked to turn red on it and green without: putting the controller back in the
tenant, giving the tenant a non-empty config.clusters, pointing a sidecar at the
wrong kubeconfig, restoring the blanket toleration, hardcoding the driver's
default owner id, dropping capmox mode, and letting the controller ignore
proxmox.csi.enabled.

The negations carry `any: true`. Without it a negated containsDocument passes
whether or not the document is in the render.

Two of the cases are about objects rather than values: the tenant release must
carry no valuesFrom at all, and two clusters must derive different volume owner
ids. Both are properties a reader can check against the rendered output, which
is why they are asserted that way rather than by reading the templates.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-29 21:04:05 +02:00
Marian Koreniuk 5ab36023e0 fix(kubernetes): release Proxmox volumes before the driver that frees them goes
Deleting a tenant cluster left its PVC disks on the hypervisor. The pre-delete
hook deletes every HelmRelease labelled with the cluster in Step 2, the CSI
driver among them, and nothing before that deleted the tenant's PVCs — a volume
is freed only by the driver answering DeleteVolume. capmox then destroys the
worker VMs, and Proxmox frees only what the destroyed VMID owns, which these
volumes are not. They stayed on storage with the tenant's data in them, and the
PV objects that could have named their owner went with the control plane.

Step 1b deletes the claims while the driver is still up, and it deletes the
claims before the Pods that hold them: a claim being deleted is one the
scheduler places no new Pod on, so a Pod that a StatefulSet or Deployment brings
back cannot mount it again and hold it to the backstop. The Pods go second: a
claim in use carries the kubernetes.io/pvc-protection finalizer, so it clears
only once nothing mounts it, and the Pods are going with the cluster in any
case. Then it waits for the PVs rather than the PVCs, because the claim
disappears as soon as its finalizer clears while the disk is freed only when the
PV is gone.

The Pods holding a claim are found by listing every Pod with the claims it
mounts and matching in the shell. A JSONPath filter on the claim name cannot do
it: with a Pod that mounts two claims its left side yields two values, client-go
refuses the whole expression ("can only compare one element at a time"), and no
Pod in that namespace would be found for any claim.

A kubeconfig that exists is not an apiserver that answers, and listing claims
against one that does not comes back empty — the clean case. Step 1b asks
/readyz first and sends an apiserver that stays silent through the backstop, so
the summary carries it. The probe is bounded by `timeout` rather than by
--request-timeout: a non-zero --request-timeout on an in-cluster kubectl call
diverts it to localhost:8080, which is why hack/check-hook-request-timeout.bats
refuses the flag in hook templates, and the check cannot tell this call, which
carries its own kubeconfig, from one that does not.

Best effort, and loud about it: a tenant that never came up has no apiserver to
ask, and a teardown must not wedge on that, but a timeout is reported through
the same backstop the other steps use.

The hook's Role gains one Secret by name — the tenant's admin kubeconfig, which
this step needs. Without it the step takes its no-kubeconfig branch and reports
that there is nothing to release while the volumes stay behind.

Step 5 no longer fails the whole hook where CDI is not installed. It deletes
DataVolumes, which are a KubeVirt-path component, and kubectl exits non-zero on
an unknown resource type; under set -e that failed the teardown after every
earlier step had already run, leaving the cluster half removed and the hook
reporting failure.

Verified on a throwaway tenant: a PVC bound and mounted by a running Pod, then
`helm uninstall`, and the volume freed by the driver rather than by hand.

delete_hook_test.yaml pins Step 1b and the Role's admin-kubeconfig entry on
proxmox and their absence on kubevirt, the /readyz probe, and the order of the
claim and Pod deletes.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-29 21:04:05 +02:00
Aleksei Sviridkin 924d28cf3d feat(kubernetes): default gpu-operator to an OS-provided driver on Talos (#3585)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

Closes the gpu-operator half of #3563 — gap 3 (the operator cannot
validate a driver it did not install) and gap 4 (nothing routes CUDA
libraries to a tenant pod).

Tenant clusters have run Talos workers since 1.6.0, where the NVIDIA
driver comes from a system extension baked into the worker image rather
than from the operator's driver container. The addon default still
described the Ubuntu shape, so a GPU node group came up with the
operator fighting the OS for the driver and no path at all for the CUDA
libraries.

Three changes, each closing one silent failure:

| Change | Without it |
|---|---|
| `driver.enabled: false`, `toolkit.enabled: false` | the driver
container loads its own module and clashes with the extension's; the
toolkit container rewrites `/etc/containerd/config.toml`, which Talos
owns and which already registers the `nvidia` runtime with `enable_cdi =
true` and `cdi_spec_dirs = ['/run/cdi']` |
| `validator.driver.env`: `DRIVER_ROOT=/`, `NVIDIA_DRIVER_ROOT=/`,
`DISABLE_DEV_CHAR_SYMLINK_CREATION=true` | the driver validator looks
for a driver-container install, finds none, never writes its sentinel,
and the device plugin stays blocked behind `toolkit-validation` — the
loop on `Attempting to validate a driver container installation` (**gap
3**) |
| `devicePlugin.env`: `DEVICE_LIST_STRATEGY=cdi-annotations`,
`NVIDIA_CTK_PATH=/usr/local/bin/nvidia-ctk`, plus `DEVICE_ID_STRATEGY`,
`NVIDIA_DRIVER_ROOT`, `CDI_ANNOTATION_PREFIX` | tenant pods carry no
`runtimeClassName` and did not need one on Ubuntu, where the toolkit
made `nvidia` the default containerd runtime. Talos does not, so CDI is
the only route: the pod gets the device and no CUDA libraries (**gap
4**) |

Every value is the configuration validated on a production Talos GPU
node group, minus the parts that only mattered while the driver
container was in play. No DaemonSet, no new image, no new package — the
fix is values.

### The removed default

The previous default fed `NVreg_NvLinkDisable=1` to the driver container
through the `nvidia-kernel-module-params` ConfigMap. With that container
off the ConfigMap is mounted by nothing, so this removes the default. On
Talos the parameter belongs to the worker machine config, which the
`kernelModules` field of the worker pool writes: that field is top-level
in the KubernetesNodes app (`packages/apps/kubernetes-nodes`), added by
#3571, and its automatic set for a pool holding an `nvidia.com/*` GPU
already loads `nvidia` with `NVreg_NvLinkDisable=1`. The wrapper chart
keeps the ConfigMap feature for an operator who re-enables the driver
container through `valuesOverride`. A test pins the absence so the dead
default cannot come back and offer a second, inert source for the same
setting.

### Relationship to #3571

Independent files, no conflict, but only useful together: this makes
`driver.enabled: false` correct, and #3571 is what actually loads the
modules the extension ships. Merged alone, this changes a GPU node group
from "operator fights the OS" to "operator stays out of the way and the
modules still are not loaded" — better, but not yet working. #3571 alone
leaves the operator fighting for the driver. Either order is safe; both
are needed for a GPU node group to work.

### What is verified, and what is not

Verified here: the rendered HelmRelease values, that `valuesOverride`
still wins over each new key, and that the removed default is gone. The
individual values are the ones running in production today, applied by
hand through `addons.gpuOperator.valuesOverride`.

Not verified here: this exact default exercised end-to-end as the
chart's own output. That needs a Talos GPU node group whose modules
load, which needs #3571 released. I would rather say so than imply a
green e2e that does not exist — happy to hold this as draft until I have
run a single-node L40S cluster against the chart default, and I will
report the result on this PR either way.

### Not covered

HAMi. When `addons.hami.enabled` the operator's device plugin stays off
and HAMi serves the GPUs; HAMi's own plugin needs equivalent CDI wiring
on Talos, which this default does not attempt. Flagging rather than
guessing — the production cluster this was derived from runs HAMi
disabled, so I have no validated configuration for that path. Worth its
own issue if a maintainer confirms the shape.

Also not in scope, and unchanged from the merge base:
`talos.installerRepository` and `talos.version` reach the worker machine
config unvalidated (being tracked separately by @lexfrei), and the
`nvidia-operator-validator` chain for the *management* cluster's own
passthrough/vgpu variants, which never load the host driver at all.

### Screenshots

No UI changes.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/643
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(kubernetes): The GPU Operator addon of tenant Kubernetes clusters now defaults to a driver provided by the operating system, which is the shape of the Talos workers these clusters have run since 1.6.0. `driver` and `toolkit` are disabled (the driver comes from a Talos system extension, and containerd configuration belongs to the machine config), the driver validator is pointed at the host root so it stops looping on "Attempting to validate a driver container installation", and the device plugin is switched to CDI (`DEVICE_LIST_STRATEGY=cdi-annotations`) so tenant pods receive the CUDA libraries — previously they received the GPU device and nothing else, since Talos does not make `nvidia` the default containerd runtime. The previous default, which fed `NVreg_NvLinkDisable=1` to the operator's driver container through a ConfigMap, is removed because that container is disabled; on Talos the parameter is carried by the worker machine config, through the `kernelModules` field of the worker pool in the KubernetesNodes app (`packages/apps/kubernetes-nodes`), which loads `nvidia` with `NVreg_NvLinkDisable=1` by default on a pool holding an `nvidia.com/*` GPU. Operators who overrode `addons.gpuOperator.valuesOverride` to work around this by hand can drop those overrides. NOTE: an override that sets `devicePlugin.env` replaces the list rather than merging it, so restate the full list when overriding it or CDI stops working. HAMi is not covered: when `addons.hami.enabled` the operator's device plugin stays off and HAMi's own plugin needs equivalent CDI wiring, which this change does not attempt.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Configuration**
* GPU workloads on Talos workers now use host-provided NVIDIA drivers
and tooling by default; the GPU Operator’s driver and toolkit are
disabled.
* Device discovery uses CDI annotations and UUID-based device IDs. When
HAMi is enabled, the Operator’s device plugin is disabled.
  * Host-root validation no longer creates `/dev/char` symlinks.
  * Driver kernel-module configuration is no longer created by default.

* **Documentation**
* Clarified when kernel-module configuration applies and what settings
are needed to re-enable the driver.
  * Noted that `valuesOverride` replaces lists wholesale.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 19:28:17 +03:00
Mattia Eleuteri 669fb2416a feat(kubernetes): default gpu-operator to an OS-provided driver on Talos
Closes the gpu-operator half of #3563. Tenant clusters have run Talos
workers since 1.6.0, where the NVIDIA driver comes from a system extension
baked into the worker image rather than from the operator's driver
container. The addon default still described the Ubuntu shape, so a GPU
node group came up with the operator fighting the OS for the driver and no
path at all for CUDA libraries to reach a tenant pod.

Three things change, each fixing one silent failure:

driver.enabled and toolkit.enabled go false. The driver container would
load its own module and clash with the extension's; the toolkit container
would rewrite /etc/containerd/config.toml, which Talos owns and which
already registers the nvidia runtime with enable_cdi = true and
cdi_spec_dirs = ['/run/cdi'].

validator.driver.env points DRIVER_ROOT and NVIDIA_DRIVER_ROOT at / and
disables /dev/char symlink creation. The driver validator otherwise looks
for a driver container install, finds none, never writes its sentinel, and
leaves the device plugin blocked behind toolkit-validation — the loop on
"Attempting to validate a driver container installation".

devicePlugin.env selects cdi-annotations and points NVIDIA_CTK_PATH at the
host's nvidia-ctk. Tenant pods carry no runtimeClassName and did not need
one on Ubuntu, where the toolkit made nvidia the default containerd
runtime; Talos does not, so CDI is the only route for the CUDA libraries.
DEVICE_ID_STRATEGY, NVIDIA_DRIVER_ROOT and CDI_ANNOTATION_PREFIX are set
and tested alongside it: CDI device names key on the UUID strategy, the
plugin resolves the driver from NVIDIA_DRIVER_ROOT, and the annotation
prefix has to be the one containerd reads, so a regression in any of them
breaks device resolution on its own.

The previous default fed NVreg_NvLinkDisable=1 to the driver container
through the nvidia-kernel-module-params ConfigMap. With that container off
the ConfigMap is mounted by nothing, so the default is removed. On Talos a
module parameter belongs to the worker machine config, which the
kernelModules field of the KubernetesNodes app
(packages/apps/kubernetes-nodes, #3571) writes. The wrapper chart keeps
the ConfigMap for an operator who re-enables the driver container.

Every value here is the configuration validated on a production Talos GPU
node group, minus the parts that only mattered while the driver container
was in play.

Not covered: HAMi. When addons.hami.enabled the operator's device plugin
stays off and HAMi serves the GPUs, and HAMi's own plugin needs equivalent
CDI wiring on Talos that this default does not attempt.

Refs: #3563

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:09:13 +02:00
Aleksei Sviridkin 2b7e0db40f fix(kubernetes): let kubevirt-csi hot-unplug from an orphaned VMI
The driver falls back to the VMI-level removevolume subresource when
a volume is still attached to the VMI but no longer in the VM spec.
The -kcsi Role grants only the VM-level subresources, and nothing
else bound to the service account covers the VMI one: kubevirt.io:
default grants no subresource of it, and the kubevirt.io:edit and
admin roles that do are aggregated only into the built-in edit and
admin roles, which the tenant namespace never binds to this account.
Every such call ends in 403, the unpublish retries until it fails,
and the hot-plug pod keeps the infra volume attached.

The call stays needed once the driver stops falling back for a
VM-owned VMI (kubevirt/csi-driver#382), because it still makes it
for a VMI no VM owns. Deleting a VM with orphan propagation leaves
its VMI running, since the KubeVirt VM controller then drops its
finalizer without stopping the VMI, so the tenant node stays up and
its volumes stay attached until the driver detaches them.

The grant adds no reach the account lacks: it can already remove a
volume from any VM in the namespace, and virt-api refuses the VMI
call on a VM-owned VMI once the HotplugVolumes gate is off.
virtualmachineinstances/addvolume stays ungranted, as the driver
only attaches through the VM.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 12:28:31 +03:00
Mattia Eleuteri 06400ac033 fix(kubernetes): probe Keycloak CRDs by output in oidc-bootstrap cleanup
`kubectl api-resources --api-group=<absent group>` exits 0 and prints an
empty list, so guarding on the exit code always fell through. The delete
then failed with `the server doesn't have a resource type "keycloakclient"`
— `--ignore-not-found` covers a missing object, not an unknown type.

Because the oidc-bootstrap Job is a post-install/post-upgrade Helm hook,
that failure blocked the Helm upgrade of every `kubernetes-<cluster>`
release on any cluster without the EDP Keycloak operator CRDs, which is
the default configuration (`oidc.mode: None`).

Capture the discovery output once and gate each delete on its own
resource type, matching the probe idiom already used by the platform
migrations and the ouroboros cleanup hook.

Fixes #3516

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
2026-09-29 09:36:55 +02:00
Aleksei Sviridkin 8dbc7bceb5 feat(kubevirt)!: drop HotplugVolumes for declarative volume hotplug
KubeVirt 1.9 deprecates HotplugVolumes in favour of
DeclarativeHotplugVolumes, which is Beta and therefore on by default,
but virt-controller skips declarative reconciliation for as long as the
old gate is on. Keeping it was only needed because kubevirt-csi
detached through the VMI-level removevolume, which virt-api accepts on
a VM-owned VMI only with the old gate. kubevirt-csi can now skip that
call, so its fallback goes off together with the gate. The two live in
different HelmReleases and roll out one after the other; in between the
driver may issue a VMI-level call virt-api rejects, and the unpublish
fails and is retried until both sides agree.

The chart now renders --enable-vmi-hotplug-fallback=false, which a
kubevirt-csi image built before the flag existed rejects at startup.
The driver image is stamped at release preparation, so this needs the
rebuilt image that ships in the same release.

DeclarativeHotplugVolumes is now listed explicitly. With HotplugVolumes
gone it is the only hotplug path, and disabling it through
disabledFeatureGates would make virt-api refuse every addvolume, so
the chart refuses that render instead.

This changes volume hotplug platform-wide. A VM-level addvolume or
removevolume now edits VM.spec directly and virt-controller reconciles
the VMI hotplug volumes against it, instead of going through
VM.status.volumeRequests. On the first reconcile it also removes every
hotpluggable VMI volume missing from VM.spec; platform migration 59
copies those into VM.spec beforehand on the variants that run
migrations.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 23:37:48 +03:00
Aleksei Sviridkin bb1b2a64f4 feat(kubernetes): tie kubevirt-csi's VMI hotplug fallback to the gate
The right value of --enable-vmi-hotplug-fallback is decided by the
KubeVirt CR in packages/system/kubevirt, not by the tenant: it must stay
true while that CR enables HotplugVolumes and turn false in the same
change that drops the gate. A note beside the gate in the CR points at
the controller Deployment, and a helm-unittest pins the flag off the
rendered args.

The chart relies on the driver default instead of rendering =true. The
driver image is stamped only at release preparation, so between
releases the chart on main runs the previous release's binary, which
does not know the flag, and Go's flag package exits on an unknown one:
rendering the default would crash-loop the controller for no effect.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 23:25:37 +03:00
IvanHunters 6602c080a1 fix(kubernetes): drop apex guard on oidc-rbac-job
The oidc-rbac bootstrap Job renders unconditionally, including the
no-etcd bootstrap path and with OIDC disabled, so guarding its
OIDC_SERVER_URL apex failed a render that must succeed: helm template
of the chart with no etcd DataStore (hack/admin-kubeconfig-invariant.bats)
aborted instead of emitting only the awaiting-etcd beacon. A running
cluster's apiserver endpoint is already guarded in cluster.yaml (etcd
present), which is the meaningful malformed-hostname case, so revert the
oidc-rbac-job guard rather than gate it further. Restore the no-etcd
unit fixture to an empty host so it keeps exercising that path.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 15:46:00 +03:00
IvanHunters 37f59369d3 fix(charts): fail fast on empty tenant apex in host composition
Charts that publish a service compose their hostname as
<sub>.<_namespace.host>. When the injected tenant apex is empty and no
per-app host override is set, they used to emit a malformed trailing-dot
hostname (e.g. "harbor.") into Ingresses, HTTPRoutes and Certificate
dnsNames silently. Source the apex through cozy-lib.ns-host-required at
each hostname-emitting site so the render fails with a clear message
instead, matching the gateway path.

Covered: harbor, postgres (external), qdrant (external), kubernetes
(control-plane endpoint), vpn, seaweedfs, bootbox, monitoring
(grafana, alerta, oidc client), and system/bucket (adds the cozy-lib
subchart symlink it lacked). The guard is placed at the point each
resource is actually emitted, so standalone renders, no-exposure
tenants and the kubernetes no-etcd beacon path are unaffected. The
post-delete cleanup hooks are left unguarded so an empty apex cannot
block uninstall.

Behavior change: a tenant with an empty injected apex and no host
override that enables one of these components now fails its render
(HelmRelease NotReady) instead of deploying a broken hostname. In
practice the apex is always injected (root has root-host, children
inherit it), so this only trips a genuinely misconfigured tenant.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 15:14:45 +03:00
Marian Koreniuk 4bc6621a5c test(kubernetes,kubernetes-nodes): assert both sides of every substrate branch
Every `if substrate` in both charts is asserted on both sides, so an inverted
condition cannot pass:

- the reconcile Job reads SVC_IP from the LoadBalancer status on proxmox and
  from the ClusterIP on kubevirt;
- the machineconfig carries the pool's nameservers on proxmox and none on
  kubevirt;
- the proxmox gate on each kubevirt-only template in the parent chart, the two
  RBAC bindings included, so a proxmox cluster renders no binding to a Role,
  ClusterRole or ServiceAccount it does not render;
- the sizing conversion at 1500m and 8G, where ceil and floor disagree
  (2 cores and 7630 MiB against 1 and 7629);
- the old-template retention loop keeps a ProxmoxMachineTemplate a MachineSet
  still references;
- the proxmox worker monitor carries no Pod threshold while the kubevirt one
  does;
- a live cluster and a live pool refuse a substrate switch while a first install
  still renders;
- both charts refuse a dnsServers entry that is not an address, the parent
  chart included, where the value reaches ProxmoxCluster.spec.dnsServers.

The negations carry `any: true`: without it a negated containsDocument passes
whether or not the document is in the render.

hack/talos-reconcile-heredoc_test.bats gains the dnsServers cases: the render
refuses a command substitution and names the entry; an entry with a colon in it
is refused too, which pins the IPv6 branch of the validation and not only the
IPv4 one; and a real address survives the Job's shell unchanged. The heredoc is
extracted by shape — `cat <<EOF` at the start of a line or right after `=$(` /
`="$(` — so the same case keeps working whichever way the Job writes it.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk c5996c1d0f fix(kubernetes,kubernetes-nodes): give Proxmox workers an endpoint they can reach
A worker on the proxmox substrate is a VM on the hypervisor network, not a Pod
in this cluster. It was handed two addresses only an in-cluster consumer can
route to: the apiserver as a ClusterIP, and the management cluster's CoreDNS
ClusterIP as its resolver. The VM booted, the Machine reported Ready with a
providerID, the disk was where it should be — and no CSR ever arrived. Nothing
in that picture points at an address.

The control-plane chart now publishes the tenant apiserver as a LoadBalancer on
the proxmox path, putting the endpoint on the network the workers share, and
the pool's reconcile Job reads the Service's LoadBalancer address instead of its
ClusterIP for the host entry the workers dial. The nameservers follow the same
reasoning: proxmox pools carry their own resolvers rather than pointing at a
CoreDNS they cannot reach, and the chart refuses to render a pool that has not
named any.

KubeVirt keeps the ClusterIP on both counts. Its workers are in this cluster,
where a ClusterIP is the right answer and a LoadBalancer address would be an
unnecessary one.

Verified end to end on a real tenant cluster: the node joins, its CSR is
approved, and cilium, konnectivity and coredns come up in the tenant.

One operational note this surfaced, not caused by it: TalosConfigTemplate.spec
is immutable, so the reconcile Job cannot rewrite an endpoint that has already
been published — the template has to be deleted for the Job to recreate it.
That matters only when an existing cluster's apiserver address changes.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk 135eb6bac4 fix(kubernetes): make the Proxmox cluster object reconcilable
Two corrections found by building a real tenant cluster on Proxmox rather than
by rendering the chart. Each produced a symptom that pointed away from its
cause.

capmox skips control-plane endpoint validation only when
spec.externalManagedControlPlane is set. Kamaji owns the endpoint here, so the
field was semantically true all along and merely absent — and on an absent
controlPlaneEndpoint the webhook does not return a validation error, it panics
(internal/webhook/proxmoxcluster_webhook.go:105). The install failed with a
nil-pointer stack trace out of an admission webhook, which reads like a broken
provider rather than a missing field.

The cluster.x-k8s.io/managed-by annotation now renders only on the KubeVirt
path. It tells Cluster API the infrastructure object is owned by something
outside CAPI, so the Cluster controller never sets an ownerRef on it — and
capmox's reconciler exits at "Waiting for Cluster Controller to set OwnerRef on
ProxmoxCluster" without ever creating the InClusterIPPool its workers draw
addresses from. Harmless on KubevirtCluster, which nothing reconciles;
load-bearing on ProxmoxCluster, which capmox must.

With both in place the object reaches Ready=True, the pool is created, and the
Cluster reaches Provisioned.

The proxmox suite's ProxmoxCluster case asserts both: the field is set and
the annotation is absent.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:14 +02:00
Marian Koreniuk db839d1198 feat(kubernetes): add a proxmox substrate for managed clusters
Companion to the same switch in kubernetes-nodes. The control plane is Kamaji
on either substrate, so this changes exactly one object — the one
Cluster.infrastructureRef points at — plus the two tenant addons that exist
only because the workers are KubeVirt VMs living in this cluster.

A KubevirtCluster needs no spec: its workers are in the same cluster and reach
the apiserver over the pod network. A ProxmoxCluster addresses an external
hypervisor, so it carries what capmox cannot infer — which Proxmox nodes may
host VMs, which addresses they may take, and who resolves names for them.
controlPlaneEndpoint stays absent on both: the cluster.x-k8s.io/managed-by
annotation hands that to Kamaji.

The address pool is required rather than defaulted. capmox has no DHCP mode; it
allocates from an InClusterIPPool built out of ProxmoxCluster.spec.ipv4Config,
so an unset pool is not a smaller configuration but a ProxmoxCluster that never
reaches Ready and worker Machines that wait on it forever. Refusing at render
time turns that into a message instead of a silence.

The KubeVirt cloud-controller-manager, its cloud-config and the kubevirt-csi
driver are gated off on the proxmox path. They reach the workers'
VirtualMachines and DataVolumes in this cluster; on Proxmox those objects do
not exist, and leaving them rendered would put controllers in the tenant
reconciling nothing. Their Proxmox equivalents are separate packages and land
separately.

The default path is unchanged: rendering it before and after this commit
differs only in the per-render random bootstrap tokens, and all 19 existing
suites pass untouched.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:29:09 +02:00
Aleksei Sviridkin 0078727620 fix(kubernetes): render the OIDC UID header only from v1.33
RemoteRequestHeaderUID is alpha and off on v1.32, so the chart had to
enable an alpha gate on v1.32 tenants with OIDC on to make
--requestheader-uid-headers start. From v1.33 the gate is beta and on
by default. No release has shipped the v1.32 path, so moving the floor
to an effective v1.33 now removes a behaviour no released chart had:
the chart renders no --feature-gates term of its own, and below the
floor it renders no UID flag and runs none of the UID checks, as it
already did on v1.31. The refusal of a list that ends in a bare flag
still applies there, because it guards the --authentication-config the
chart appends on every version.

The refusal of a kube gate spelling mixed between bare and
kube:-prefixed terms goes too. Once the chart writes no gate term, the
spelling of the operator's terms no longer concerns anything it
renders, and the chart already passes other gate values the apiserver
refuses, unknown names and unparseable values, through to it.

--emulated-version is still read, because emulation moves the
effective version below the floor. Its kube version is now read from
the numbers at the start of the value, with the element split at its
first equals sign as toVersionMap splits it, so v1.32, 1.32.0 and
1.32-rc all read as 1.32. With OIDC on, an --emulated-version entry
without quotes whose kube version has a non-zero patch component fails
the render and names the entry: the apiserver refuses such a value and
would otherwise only crash-loop.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 15:10:53 +03:00
Aleksei Sviridkin 4e9f83ac0d fix(kubernetes)!: move or refuse extraArgs flags the control plane owns
From 26.9.4-edge the tenant control plane sets a list of kube-apiserver
flags itself and drops any extraArgs entry naming one before the
apiserver sees the list; 26.3.6-edge let such an entry win. An existing
tenant that set, for example, its own admission plugins through
extraArgs would lose them on the first reconcile after the upgrade with
nothing to say so, and a flag written with its value in a separate
entry leaves that value behind as a positional argument the apiserver
refuses.

Two of those flags have a KamajiControlPlane field the control plane
renders them from. An --enable-admission-plugins=value or
--kubelet-preferred-address-types=value entry is moved into that field
while the field is empty, and left out of the rendered extraArgs, so
the tenant keeps its setting across the upgrade without an edit. The
move is computed from the values on every render; the last entry wins,
as it did while the control plane kept one entry per flag name, and a
repeated address type is dropped because that field is a set. The
post-upgrade Job that patches extraArgs onto the KamajiControlPlane
reads the same list.

Every other entry naming an owned flag is refused at render time in
every OIDC mode, since the chart cannot stop the drop and the refusal
is where the operator learns of it. So are the two moved flags beside a
non-empty field, with a value that names nothing or is quoted, and in
the two-entry spelling. The list mirrors the Kamaji builder for the
shape of control plane this chart renders, and the extraArgs
description names the same flags.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 14:59:12 +03:00
Aleksei Sviridkin 0f9b8dc16c feat(kubernetes): expose admission controllers and kubelet address types
The KamajiControlPlane carries two lists the tenant control plane
renders into kube-apiserver flags: spec.admissionControllers into
--enable-admission-plugins and spec.kubelet.preferredAddressTypes into
--kubelet-preferred-address-types. The chart rendered neither from the
tenant values; the address types were a fixed list. Pass both through.

Admission controllers are rendered only when the list is non-empty, so
a tenant that sets none keeps the control plane's own default set, and
a list replaces that default rather than adding to it. An empty or
absent list of address types renders the list the chart rendered
before, so a tenant that sets nothing renders exactly as it did.

The names are declared as enums copied from the KamajiControlPlane
CRD at the provider pin, so the application API refuses a misspelled
name where the tenant edits it instead of at apply time. Both fields
are optional.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 14:59:11 +03:00
Aleksei Sviridkin 5b370d533f fix(kubernetes): read every extraArgs entry for the OIDC UID flags
The tenant control plane hands the kube-apiserver every entry of a flag,
so reading only the last one misses an opinion the apiserver acts on. An
earlier --feature-gates entry turning RemoteRequestHeaderUID off, or an
earlier --emulated-version naming kube, was previously dropped before the
apiserver saw it; now it decides, and rendering the uid header flag
against the later entry alone stops the control plane from starting.

Repeats of --requestheader-uid-headers append, so the chart adds its own
element beside a list that omits X-Remote-Uid instead of refusing to
render: the union satisfies the check the refusal existed for.
--feature-gates terms accumulate into one component bucket, so the chart
carries only its own term rather than a copy of the operator's, and
naming the kube component both bare and kube:-prefixed is refused
outright, which nothing the chart renders can repair.

A flag and its value written as two entries is read the same way the
apiserver reads it. pflag pairs them, so the value belongs to an entry
this code reads separately: the uid header flag counts as present, which
is what the gate-disabled check needs, while --feature-gates and
--emulated-version decide what is rendered and are not guessed at. What
holds whatever such a value hides still applies next to one: an empty
header element is refused, an explicit opt-out beside a header entry is
refused unless the hidden value is a gate term that could undo it, and
X-Remote-Uid is added beside an operator's own header list, which it can
only help. The chart's other additions wait for a reading.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 14:58:12 +03:00
Aleksei Sviridkin e2464ea788 fix(kubernetes): give System-mode tenant OIDC users a UID
The AuthenticationConfiguration the chart renders mapped username and
groups only, so upstream resolved every System-mode user to an empty
UID and the X-Remote-Uid header the tenant apiserver publishes reached
an aggregated API server carrying nothing.

Map uid to the sub claim. The platform realm enables Keycloak's
UPDATE_EMAIL required action, so the address the username claim
carries is mutable and cannot serve as a stable identifier, while sub
is the account UUID and survives a change of either. CustomConfig
mode is untouched: an operator-supplied configuration is still
written verbatim, uid mapping included or not.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-22 23:04:28 +03:00
Aleksei Sviridkin d870b392fa fix(kubernetes): render the aggregation-layer UID header for tenant OIDC
A tenant kube-apiserver publishes requestheader-uid-headers into the
extension-apiserver-authentication ConfigMap only when the flag is set,
and the chart never set it. Without that key the request-header
authenticator on an aggregated apiserver does not know which header
carries the UID and ignores it, so a caller who has a UID is served
without one. Operators had to add the flag by hand through the
controlPlane.apiServer.extraArgs escape hatch.

This forwards a UID the caller already has and does not create one.
ServiceAccount tokens carry a UID and gain from it. OIDC users
authenticated through spec.oidc.mode: System do not: the
AuthenticationConfiguration the chart generates maps username and
groups only, and with no claimMappings.uid upstream returns an empty
UID (plugin/pkg/authenticator/token/oidc/oidc.go, getUID, read at
v0.35.0), so there is nothing to carry. Mapping a claim would give
those users a UID and change the identity of users who have none
today, which is a separate decision and is tracked separately.

The half read rather than run: that an existing UID is lost without the
published key comes from the controller writing the ConfigMap and the
request-header authenticator consuming it, not from a live cluster. That
OIDC users have no UID is unambiguous in the source named above.

Render --requestheader-uid-headers=X-Remote-Uid whenever OIDC is on,
plus --feature-gates=RemoteRequestHeaderUID=true on v1.32, where the
gate is still alpha and the apiserver rejects the flag without it.
v1.31 gets nothing: the flag does not exist there and an unknown flag
is fatal to the apiserver.

The deciding version is the effective one, not the image tag.
--emulated-version in the same passthrough moves it down, and the gate
resolves at the emulated version through featureSpecAtEmulationVersion
(featureSpecAtEmulationAndMinCompatVersion by v1.35), so an emulated
1.32 needs the gate line even on a v1.35 cluster, and below 1.32 the
gate is PreAlpha and cannot be enabled at all, which leaves nothing
safe to render.

Both flags are new, so every extraArgs shape reaching this code was
rendering before it existed, and turning one of those into a render
error would strand a running tenant on a HelmRelease that stops
converging while its control plane carries on. The chart therefore
yields to an operator entry that already satisfies the requirement,
renders nothing at all when the operator turned RemoteRequestHeaderUID
off, and otherwise merges its gate term into a copy of the operator's
--feature-gates value. Carrying that value into the merged entry
decides the gate whether the tenant control plane collapses duplicate
flags into the last one or hands both to the apiserver, which merges
them in the order it reads them.

Operator entries are read the way the apiserver reads them: header list
elements matched whole and untrimmed, folded as EqualFold folds them,
and refused when one is empty after trimming, as checkForWhiteSpaceOnly
refuses it; gate names compared exactly; gate values resolved as
strconv.ParseBool resolves them; blanket AllAlpha and AllBeta honoured;
an empty value treated as no entry at all. Each flag is judged by its
last entry carrying a value. The apiserver's CSV reader strips quotes
the chart does not parse, so quotes on that entry make the chart render
nothing, while quotes on an earlier entry the collapse drops do not.

The render fails only where a uid-headers list of the operator's
describes a control plane that already refuses to start: one that
omits X-Remote-Uid, one with an element that is empty after trimming,
and any such list while the gate is off. Withholding the chart's own
flag rescues none of them, because the flag the apiserver rejects is
the operator's.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-22 18:23:48 +03:00
Alexey Artamonov a3df02df23 fix(kubernetes): name the configured realm in the OIDC denial message
The claimValidationRules message in the System-mode AuthenticationConfiguration
hardcoded "in the cozy realm", so on a renamed realm it told a rejected user to
look in a realm that no longer exists. Build it from the same realm the issuer
uses (toString-guarded, as the derived issuer URL already is).

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-21 12:15:10 +03:00
Alexey Artamonov 44a9a8e735 feat(platform): make the Keycloak realm name configurable
The realm name "cozy" was hardcoded in keycloak-configure (which owns
the realm, its clients and the cozystack-cluster-admin roles), in the
dashboard and linstor-gui gatekeepers, in the info kubeconfig, and in the
Grafana and managed-Kubernetes apiserver OIDC issuers. Operators could
not rename the realm without patching vendored templates, which
`make update` reverts. The apiserver OIDC issuer is already configurable
via talm oidcIssuerUrl, so pointing it at a custom realm left every SSO
component on "cozy" and broke login with an issuer mismatch.

authentication.oidc.realmName (default "cozy") now rides the _cluster
channel so every consumer resolves the same realm, and the default
keycloakInternalUrl is derived from it so the realm is named in one place.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-21 12:15:10 +03:00
Aleksei Sviridkin 355630a5e7 fix(kubernetes): pin KubeVirt CSI sidecars to SIG Storage releases (#3934)
## What this PR does

Replaces five mutable OpenShift `latest` image references (four unique
sidecar images) in the KubeVirt CSI controller and tenant-node DaemonSet
with versioned Kubernetes SIG Storage releases. This addresses the
affected references from #3578 without claiming the other references in
that issue.

The selected controller versions match the sidecars already used by
Cozystack's LINSTOR CSI deployment. Render tests pin every changed
container name and image so a mutable tag cannot return silently.

Validation:

- `helm unittest packages/apps/kubernetes`: 247/247 passed.
- `helm unittest packages/system/kubevirt-csi-node`: 6/6 passed.
- Live MSK-prod guest-cluster canaries on 2026-08-20 and 2026-08-21
reproduced the failure with the stock chart: the `latest` sidecars
exited on the old CPU with the x86-64-v3 glibc error and the controller
stayed in CrashLoopBackOff. A disposable guest cluster with the pinned
images reached a 7/7 Ready controller on 2026-08-19 and was removed
after the check.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the four changed template/test files. No
package, schema, API, default, CRD, tooling contract, or downstream
hardcoded value changes.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(kubernetes): pin KubeVirt CSI sidecars to versioned SIG Storage images that run on pre-x86-64-v3 CPUs
```
2026-09-18 03:48:47 +04:00
Myasnikov Daniil 3daa83f4b5 feat(cilium): surface a kube-proxy-replacement override instead of nothing
The pin added earlier guards a cozystack developer changing the default. It
cannot observe an operator override in the field, and neither can the e2e,
which only ever runs the default configuration -- so the path Ivan's review
actually names, addons.cilium.valuesOverride on a tenant cluster and the
cozystack.networking Package's cilium component on the host, produced no signal
at all. Both webhook deny policies name their webhook's backend port only,
which covers a pod dialling the ClusterIP solely because Cilium performs the
service DNAT ahead of its own egress hook, so an override degrades that half
silently.

A status-beacon ConfigMap rather than a fail(), following the -awaiting-etcd
beacon in packages/apps/kubernetes/templates/cluster.yaml. Whether Cilium
without kube-proxy replacement is a supported cozystack configuration is a
platform decision, and refusing to render the CNI over a weakened
defence-in-depth control is not a call these charts should make. The host
beacon fires on a bool false, on the string "false", and on the umbrella key
being deleted, because the subchart's own "false" default coalesces into the
parent. The tenant beacon fires on an explicit non-true override only, since
absence means the chart default applies, and it renders on the management
cluster next to the HelmRelease carrying the override -- which is why the
mismatch is detectable there at all rather than inside the guest cluster.

Also withdraws the argument the suite header used to carry, that a target-side
ingressDeny could not be re-added as a fallback. It was wrong, and the next
commit adds that rule. The header now records what the two halves each buy.

Makefile gets .PHONY: test for the reason this branch already argued for
kubeovn-webhook, which applies with more force here now that the suite is
load-bearing for two other packages: hack/package.mk's .PHONY list deliberately
carries only the targets that file defines, so a path named `test` would
otherwise make the target up to date and skip the recipe. Proven both ways.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-02 20:44:53 +05:00
Yan Bondarenko 296d3008d9 fix(kubernetes): pin KubeVirt CSI sidecars to SIG Storage releases
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-08-21 15:51:53 +03:00
IvanHunters 69da032103 docs(kubernetes): point references at migration 56 after the renumber
The merge renumbered the worker-pool adoption migration from 54 to 56, but
six references still named 54 (README, pre-delete-unpin hook, cluster.yaml,
worker_adoption_guard test), which now points at main's redis-operator
adoption migration. Repoint them at 56.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-21 00:35:05 +03:00
IvanHunters 7a9226da15 test(kubernetes): make control-plane-only negative asserts actually check
The 'emits no worker-pool objects' assertions used 'not: true' with
'documentIndex: -1', which scopes the check to the last rendered document
only, so a stray MachineDeployment/KubevirtMachineTemplate/MachineHealthCheck/
TalosConfigTemplate anywhere else passed silently. Move 'any: true' inside
the containsDocument matcher and drop documentIndex so the negation covers
every document. Verified the asserts now redden when a worker object is
present.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-21 00:35:05 +03:00
IvanHunters f035da8a0f Merge branch 'main' into feat/kubernetes-nodes-phase2b
Resolve conflicts introduced by main:
- Drop the FluxCD tenant addon (API type, deepcopy, values schema, RD) in
  favour of the GPU type main added; main removed the fluxcd addon wholesale.
- Renumber the Phase 2 worker-pool adoption migration 54 -> 56, behind main's
  redis-operator (54) and fluxcd-removal (55) migrations; targetVersion 55 -> 57.
- Accept the branch's removal of the in-chart talos-reconcile Job; it moved to
  the kubernetes-nodes chart, main only touched its comment.
- Regenerate kubernetes values.schema.json / README and the kubernetes-rd
  ApplicationDefinition schema.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-20 14:04:32 +03:00
IvanHunters fd8d378b73 fix(kubernetes): rotate bootstrap-token Job on either token half; review nits
Fold bootstrapTokenSecret into the Job content hash alongside
bootstrapTokenId. talos-secrets.yaml regenerates the two token halves
independently, so hashing only the id left a rotated secret no-oping a
completed Job and the tenant stuck on the stale bootstrap-token.

Also address review feedback: drop a review-round reference from the
tracked test, correct a stale hash in a test comment, pin the
alternate-image hash case exactly instead of by regex, and fix a comment
indent in talos-reconcile-job.yaml.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-20 11:56:44 +03:00
IvanHunters 035bde1920 fix(kubernetes): correct bootstrap-token Job rationale and make it churn-free
Address review of the main-phase bootstrap-token Job.

The deadlock the header claimed cannot occur on this release: the
kubernetes cozyrd carries helm-install-disable-wait, which cozystack-api
maps to Install.DisableWait and Upgrade.DisableWait, so there is no
main-resource wait phase in front of the hooks. Rewrite the rationale
around the real reason to leave the hook shape: a post-upgrade hook never
runs on a release whose upgrade keeps failing, so the token Secret never
lands. Fix the stale 'stays a post-install hook' note in
talos-reconcile-job.yaml and the 'fails the chart upgrade loudly' comment,
which is untrue once Helm no longer waits on the Job.

Give the immutable Job a content-hash name suffix over its whole rendered
spec, mirroring talos-reconcile: a static name would fail an upgrade that
lands while the Job is live (or inside its TTL) with 'spec.template: field
is immutable'. Fold the looked-up bootstrapTokenId into the hash so a
rotated token also rolls the Job instead of Helm no-oping a completed one.

Make the admin-kubeconfig volume optional and gate the Pod on the shared
wait-for-kubeconfig init container, so it waits in init (uncounted by
backoffLimit) instead of FailedMount at t0.

Extend the template tests for the hashed name, its rotation on an image
change, and the optional volume + init container.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-20 11:56:44 +03:00
IvanHunters 542ab899c2 fix(kubernetes): break bootstrap-token Job out of Helm hook to unblock fresh installs
The bootstrap-token-tenant Job was wired as a Helm post-install/post-upgrade
hook. post-* hooks fire only after Helm's wait phase resolves, and the tenant
kubernetes release wait blocks on MachineDeployment Ready, which blocks on
nodes registering, which needs the kubelet CSR path — for which the very
bootstrap-token this Job produces has to already exist in the tenant
kube-system. Fresh installs and any HR reconcile that finds MD in a
non-Ready state deadlock; HR.upgradeFailures climbs while post-upgrade
hooks never get their turn to run.

pre-install/pre-upgrade would not help either: they fire before non-hook
templates apply, so <release>-admin-kubeconfig (materialised by Kamaji
after KamajiControlPlane is applied) is not yet available.

Move the ServiceAccount and Job to the main install phase, matching what
talos-reconcile already does in this chart. Helm applies them together
with the KamajiControlPlane and talos-secrets; the Job Pod stays pending
until Kamaji publishes the admin kubeconfig Secret (backoffLimit covers
the retry window) and then completes before MachineDeployment can
converge. Helm's wait treats the Job as another resource to reach
terminal state and does not gate it on MD.

Extend the existing template test to pin the absence of helm.sh/hook
on both the ServiceAccount and the Job, so this cannot regress silently.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-20 11:56:43 +03:00