mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 20:23:39 +09:00
f83db514b0eb88cefc2923d17aad59daccb4e358
181
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
95dc591198 |
test(kubernetes): keep application identity out of guest log streams
Guest pods are tenant-controlled, so their lineage labels are not a trustworthy application identity, and the platform must not promote them to the stream fields the parent scopes application logs by. The guest release replaces the monitoring-agents filters and outputs wholesale, which is what keeps the parent's label lift out of guests; pin that so a later change to the guest overrides cannot bring it in. This only covers what the platform ships. A guest admin can still edit the guest's fluent-bit or write to the parent's vlinsert directly; that needs a write-path control, which the logs proposal leaves open. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
ae9e7c4c64 |
test(kubernetes): pin RetryOnFailure on the Proxmox csi release (#4969)
The KubeVirt csi and cilium releases carry a helm-unittest pin on `strategy.name: RetryOnFailure` for both actions, and the e2e HelmRelease guard in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` names those two tests as load-bearing. `helmreleases/csi-proxmox.yaml` sets the same strategy for the same reason and had no pin. This adds `packages/apps/kubernetes/tests/csi_proxmox_install_retry_test.yaml`, a mirror of `csi_install_retry_test.yaml` rendered on the proxmox substrate, and updates the guard comment that pointed at the gap. No template change. Verified with `helm unittest .` in `packages/apps/kubernetes` (32 suites, 349 tests). Closes #4731 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Expanded automated coverage for Proxmox CSI installation and upgrade retry behavior, including the retry interval and remediation retry limits. The checks verify that retries continue according to the configured settings during installation and upgrades. This update does not change user-facing functionality. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
b43196e549 |
fix(kubernetes): pin the Proxmox cloud-config builders to the bundled image
The Proxmox substrate added two init containers that build the cloud config from the hypervisor token Secret, and both still took their image from images.waitForKubeconfig. With that key gone the render failed on a nil map; with it set, a tenant chose the image that reads the token. They now run the bundled busybox through cozy-lib.image, like the wait-for-kubeconfig container next to them. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
1636349a63 |
fix(api): point the removed image overrides at node registry mirrors
The warning for a removed image override told an air-gapped install to use the platform-wide registry, but nothing in the platform writes _cluster.images-registry, so there is nothing to set. What serves these images offline is what serves every other platform image: the registry mirrors configured on the nodes, as the air-gapped install guide sets up. The tests that hold the cozy-lib.image routing now say what they check rather than promising a platform knob. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
dfff5c2dac |
test(kubernetes): hold the routing of the busybox and unpin images
The busybox helper shared by four workloads and the pool's pre-delete Job had no test holding their route through cozy-lib.image, so reverting either line left both suites green. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
d99a1b9f2f |
fix(kubernetes): route the helper images through cozy-lib.image
These images now go through cozy-lib.image like the harbor, kafka and mariadb hook images, so they follow _cluster.images-registry wherever that is set rather than any per-application value. An air-gapped install serves them the way it serves every other platform image, through the registry mirrors configured on its nodes. The talos-csr-signer sidecar and the OIDC Jobs had no test holding their image, which is where an override could come back unnoticed. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
af20694359 |
fix(kubernetes)!: drop the helper image overrides
images.waitForKubeconfig, images.kubectl and images.talosCsrSigner ran whatever image a tenant named inside the management cluster: the csr signer as a sidecar of the tenant control plane with the Talos CA key mounted, the other two in pods holding service account tokens for CAPI, KubeVirt and Cilium objects in the tenant namespace. A tenant never gets that access through the apps API, and the platform should not hand it over through an image field. The images now always come from the bundled tag files. A release that still carries the keys renders exactly as one without them. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
98d0402096 |
test(kubernetes): pin RetryOnFailure on the Proxmox csi release
The KubeVirt csi and cilium releases carry a helm-unittest pin on strategy.name: RetryOnFailure for both actions, and the e2e HelmRelease guard names those two tests as load-bearing. helmreleases/csi-proxmox.yaml sets the same strategy for the same reason and had no pin. Add the mirror test and update the guard comment that pointed at the gap. No template change. Closes #4731 Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
904bc38fc3 |
fix(kubernetes): delete the Talos PKI and oidc-bootstrap Job on uninstall
cert-manager runs with enableCertificateOwnerRef: false, so the <release>-talos-ca and <release>-talos-tls-cert Secrets carry no ownerReference to their Certificates and outlive the release, keeping the CA private key of a cluster that no longer exists. A new cluster created under the same name then adopts that stale CA instead of issuing its own trust root. The <release>-oidc-bootstrap Job is a hook whose only delete policy is before-hook-creation. Hooks are not part of the release manifest, so uninstall never removes it. The pre-delete hook deletes the two Certificates before their Secrets, because deleting a Secret while its Certificate exists makes cert-manager re-issue it. It runs after the TenantControlPlane is gone, since the control plane's CSR signer mounts the CA. The Role grants each delete by resourceName only. Assisted-by: LLM Signed-off-by: Ray Low <crossrt@gmail.com> |
||
|
|
5486d4ffcf |
fix(networking): harden webhook policy assumptions (#4058)
## What this PR does This is follow-up for [#3997](https://github.com/cozystack/cozystack/pull/3997). It reads ingress-nginx admission port from effective subchart values and refuses empty value, and fixes `kubeProxyReplacement=false` status because cozystack has no supported Service translator in that state. Gateway test now requires one real ClusterIP and keeps same ready Pod UID, IP and port for whole probe, so stale EndpointSlice, replaced Pod or wrong Service can't report deny. Policy selectors and two-ended deny from [#3997](https://github.com/cozystack/cozystack/pull/3997) stay unchanged, and this does not claim root cause for [#3983](https://github.com/cozystack/cozystack/issues/3983). ### Screenshots Not applicable. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(networking): use effective ingress admission port, report unsupported Cilium service routing correctly, and reject stale backend evidence in gateway tests ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Admission webhook connectivity checks now validate the configured Service and its exact backend endpoint throughout probing, improving failure detection. - Admission webhook policies now fail closed when the webhook port is not explicitly configured. - **Documentation** - Clarified that supported service routing requires `kubeProxyReplacement: true`. - Updated guidance to explain that disabling this setting or webhook policies does not provide a supported fallback. - **Tests** - Added coverage for missing webhook port configuration and strengthened networking stability checks. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
c9558f5658 |
fix(networking): correct webhook policy assumptions
Read the ingress admission port from the effective subchart values and fail if the port is absent. Describe kube-proxy replacement as a required service-routing invariant while retaining target-side enforcement. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
520af57905 |
chore: drop private cluster, host and personal names from the tree
Private infrastructure names, a real hostname, a commented-out internal mirror block, an internal product name, and a reviewer's name in a comment do not belong in a public repository; drop them and replace what needs a stand-in with neutral placeholders. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
bcda5c893a |
feat(kubevirt)!: drop HotplugVolumes for declarative volume hotplug (#4453)
## What this PR does This turns off the deprecated KubeVirt `HotplugVolumes` gate and moves the platform to declarative volume hotplug. It changes how every volume hotplug on the platform works, so the title carries `!`. Closes #4032. This PR is stacked on #4451 and has to merge after it. #4451 adds the kubevirt-csi flag this PR flips. The replacement gate is already on in KubeVirt 1.9, and the old gate is the only thing keeping it off. `DeclarativeHotplugVolumes` is Beta at v1.9.0, and Beta gates are on by default. virt-controller skips `handleDeclarativeVolumeHotplug` while `HotplugVolumes` is on, so dropping the old gate is what switches the new one on. The kubevirt-csi fallback goes off in the same change. With the old gate off, virt-api rejects the VMI-level `removevolume` on a VM-owned VMI, so the chart now passes `--enable-vmi-hotplug-fallback=false`. The comments next to the gate and the flag point at each other. This needs the rebuilt kubevirt-csi image. A driver built before #4451 does not know the flag and exits on it at startup. The image is stamped at release preparation and ships in the same release as this change, so only a `make apply` from main before then would crash-loop the CSI controller. What changes at runtime on an upgrade: - The VM-level `addvolume` and `removevolume` calls now edit `VM.spec` directly. virt-controller then makes the VMI hotplug volumes match it, instead of going through `VM.status.volumeRequests`. - Volumes hot-plugged through the VM subresource are in `VM.spec`, so they stay attached. - During a tenant teardown, when the VM goes before its VMI, the driver no longer calls the VMI subresource. It waits for the VMI status to drop the volume, and the garbage collector deletes the VMI with its VM. Upstream waits up to two minutes. If the VMI is still there after that, the unpublish fails and is retried. A new platform migration keeps old VMI-only volumes attached. On its first pass virt-controller removes every hotpluggable VMI volume that is missing from `VM.spec`. Until July 2025 kubevirt-csi attached volumes at the VMI level (the Role then granted `virtualmachineinstances/addvolume`), and live migration carries such a VMI across KubeVirt updates. Without help, a tenant node not recreated since then would lose a mounted disk on upgrade. Migration 59 copies each such volume and its disk from the VMI into the VM template before anything else changes. Adding a hotplug volume to the VM spec is a valid live update in KubeVirt, so no VM needs a restart. It runs as the platform chart's pre-upgrade hook, before the upgrade re-points `cozystack-packages`. The KubeVirt CR and the tenant kubernetes charts both render from that source, so the gate and the flag cannot change before the migration has finished. It only touches VMs that need it, logs each one, and a second run changes nothing. The migration also makes ephemeral hotplugs permanent. A volume a user added at the VMI level, for example with `virtctl addvolume` without `--persist` from an older virtctl, has the same shape as kubevirt-csi's old attachments, because virt-api builds both the same way. There is nothing reliable to tell them apart by, so both are copied. Leaving the user's volume out would not keep it ephemeral: the first reconcile would unplug it from the running VM. After the upgrade it is part of `VM.spec` and attaches again on a restart, and `virtctl removevolume` drops it. Ephemeral hotplug stops working on VMs. With the old gate off, virt-api refuses VMI-level `addvolume` and `removevolume` on a VMI that a VM owns (`vmiVolumePatch` in v1.9.0). A volume on such a VMI is added and removed through the VM from then on. A VMI without a VM is not affected. Only running VMIs are copied. virt-controller reconciles no other VMI, and a stopped VM builds its next VMI from `VM.spec`, so copying from a stopped one would bring back volumes the restart was going to drop. Migrations run only on the isp-full, isp-full-generic and isp-hosted variants. KubeVirt comes from the platform only on isp-full and isp-full-generic, so those are covered. On the default variant, where packages are installed by hand, nothing runs the migration. This optional check lists the VMIs and volumes the migration would copy. It uses the same rule. On the default variant, run it before upgrading and recreate each tenant node it lists: ```bash kubectl get vm,vmi --all-namespaces --output json | jq --raw-output ' ([.items[] | select(.kind == "VirtualMachine") | {key: "\(.metadata.namespace)/\(.metadata.name)", value: [.spec.template.spec.volumes[]?.name]}] | from_entries) as $vm | .items[] | select(.kind == "VirtualMachineInstance") | ([.metadata.ownerReferences[]? | select(.controller == true and .kind == "VirtualMachine" and (.apiVersion | startswith("kubevirt.io/"))) | .name][0]) as $owner | "\(.metadata.namespace)/\($owner)" as $key | select(.status.phase == "Running" and $owner != null and $vm[$key] != null) | .spec.volumes[]? | select(.dataVolume.hotpluggable == true or .persistentVolumeClaim.hotpluggable == true) | .name as $n | select($vm[$key] | index($n) | not) | "\($key) \($n)"' ``` Empty output means nothing is affected. I checked the query against the migration's test fixtures only. Neither the query nor the migration has run on a live cluster. #4287 also adds a migration 59. If it merges first, this one moves to 60: the file, the version it stamps, `migrations.targetVersion`, the bats file (its `run_migration` argument and expected `STAMP`) and `hack/testdata/migration-59-vmi-hotplug`. A VM with a pending hotplug request at upgrade time may also get stuck. virt-controller still processes `VM.status.volumeRequests` before the declarative handler, and it applies them through the VMI subresource, which virt-api rejects once the old gate is off. The sync then fails before the VM spec is updated, so nothing clears the request, and every later sync of that VM fails the same way. A retry through the VM subresource edits the spec and clears it, which covers kubevirt-csi's own requests. A one-shot request stays stuck until the VM restarts. I read this from the v1.9.0 code and haven't reproduced it. Waiting until this lists nothing before upgrading narrows the window, but a request made during the rollout can still hit it: ```bash kubectl get vm --all-namespaces --output json | jq --raw-output '.items[] | select((.status.volumeRequests // []) | length > 0) | .metadata.namespace + "/" + .metadata.name' ``` The two halves live in different HelmReleases, so there is a short window during the rollout where the driver and the gate disagree. In that window an unpublish fails and is retried until both are updated. A tenant cluster whose `kubernetes` HelmRelease is suspended or failing keeps its old driver for longer. Drivers from before July 2025 attach through the VMI subresource, so on such a tenant every attach is refused until its release reconciles. Such a driver can also attach a volume after the migration has run and before the gate flips. The migration doesn't see that volume, and the first declarative reconcile unplugs it from the running VM. The migration lists the VMs once and patches each one with a `resourceVersion` test, so a VM that changes in between fails the hook Job. The Job has `backoffLimit: 3`, and each retry lists again and skips the VMs already patched. Re-reading only the conflicting VM would avoid the retry; I left that out. The new gate is listed in the CR even though it is on by default. With the old gate gone it is the only hotplug path, and the chart refuses a render that disables it. I found no other user of `HotplugVolumes` in the platform or in the downstream repositories. The VMI-level fallback may have been failing already. The kcsi Role has no rule for the VMI-level `removevolume` (#4452), so that call likely got 403 with or without the gate. I haven't checked that on a cluster. `hack/migration-59-vmi-hotplug-volumes.bats` runs the migration against a fake kubectl: a VM that needs the copy, VMs and VMIs that must be left alone (a VMI that is not Running among them), a second run, a cluster without KubeVirt and the fail-closed paths. The helm-unittests for the KubeVirt CR pin the whole gate list, accept `HotplugVolumes` in `disabledFeatureGates` and refuse `DeclarativeHotplugVolumes` there. The CSI test pins the flag at false, and the `make update` idempotency test still passes. None of that exercises hotplug itself. The real test is e2e with volume attach and detach on a tenant cluster, including a teardown. No CI lane upgrades from the previous release yet, so an upgrade with a live VMI-only volume is untested; #4463 tracks that test. ### Screenshots No UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(kubevirt)!: KubeVirt now runs with DeclarativeHotplugVolumes instead of the deprecated HotplugVolumes feature gate, and kubevirt-csi no longer uses the VMI-level hot-unplug. Volume hotplug requests are applied through VM.spec. The upgrade copies volumes that older kubevirt-csi versions attached only to the VMI into VM.spec, so they stay attached; the check in the PR description is optional. This also applies to ephemeral volumes added to running user VMs at the VMI level, such as with virtctl addvolume without --persist: they become part of VM.spec and attach again after a VM restart. virt-api now refuses VMI-level addvolume and removevolume on a VMI owned by a VM, so such volumes are added and removed through the VM. A tenant cluster whose kubernetes release is suspended or failing keeps its old kubevirt-csi, and a driver from before July 2025 cannot attach volumes there until that release reconciles, and a volume it attaches during the upgrade itself can be unplugged again. A values override that puts DeclarativeHotplugVolumes into disabledFeatureGates now fails the kubevirt render. Clusters on the default variant do not run platform migrations and should run that check before upgrading. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Declarative hotplug volumes are enabled by default for KubeVirt. * Running virtual machines keep eligible hotplugged volumes and disks when their configuration is migrated. * **Bug Fixes** * The migration avoids adding duplicate volumes or disks and does not mark the migration complete if required operations fail. * The Kubernetes CSI driver no longer uses the VMI hotplug fallback. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
86bf2ba8ec |
fix(kubernetes): let kubevirt-csi hot-unplug from an orphaned VMI (#4563)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does This lets the kubevirt-csi controller detach a volume from a VM instance that no VM owns. Right now that call is always denied. Closes #4452. The driver has a fallback for a volume that is still attached to the VMI but is no longer in the VM spec. It calls the VMI-level `removevolume` subresource. The `-kcsi` Role grants only the two VM-level subresources, so the call ends in 403. Nothing else reaches this service account with that grant. Only its own Role and ClusterRole name it as a subject, and the group bindings that cover every service account grant nothing on this subresource. At v1.9.0, `kubevirt.io:default` grants only `kubevirts`, `version` and `guestfs`. `kubevirt.io:edit` and `kubevirt.io:admin` do include the VMI subresource, but they aggregate only into the built-in `edit` and `admin` roles, and no tenant binding gives those to this account. The cozystack tenant roles aggregate on their own labels and don't list it. The fallback is still needed after the flag in #4453 goes to false. The flag only skips VMIs that a VM owns. When a VM is deleted with `propagationPolicy=Orphan`, the KubeVirt VM controller drops its finalizer and leaves the VMI running (`vm.go` at v1.9.0). The tenant node stays up with its volumes attached. When a pod moves off that node, the driver has to detach through the VMI, and on a 403 the hot-plug pod keeps holding the infra volume. Until #4453 lands, the fallback also runs for VM-owned VMIs and gets the same 403. The grant is `update` on `virtualmachineinstances/removevolume` and nothing more. The account can already remove a volume from any VM in the namespace, so it gets no new reach. virt-api also rejects the VMI-level call on a VM-owned VMI once `HotplugVolumes` is off. `virtualmachineinstances/addvolume` stays out because the driver only attaches through the VM. The new helm-unittest pins the Role's `subresources.kubevirt.io` rule to these three subresources with `update`. I haven't reproduced the 403 on a cluster. This shows it on a tenant namespace, before and after: ```bash kubectl auth can-i update virtualmachineinstances.subresources.kubevirt.io --subresource=removevolume \ --namespace <tenant-ns> --as system:serviceaccount:<tenant-ns>:<release>-kcsi ``` Upstream kubevirt/csi-driver has the same gap in `deploy/infra-cluster-service-account.yaml`. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> No UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(kubernetes): the kubevirt-csi controller can now hot-unplug a volume from a VMI that no VM owns, instead of failing with 403 Forbidden. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved Kubernetes CSI volume removal for virtual machine instances, including cases where a volume is not listed in the virtual machine specification. This helps the driver complete volume removal in scenarios where the Kubernetes virtualization API accepts the instance-level request. Existing permissions for adding and removing volumes from virtual machines remain unchanged. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
94abf6940b |
feat(kubernetes): let kubevirt-csi detach without the HotplugVolumes gate (#4451)
## What this PR does This lets kubevirt-csi detach volumes without the deprecated KubeVirt `HotplugVolumes` gate. The default stays as it is, so nothing changes at runtime yet. Part of #4032. The driver got a switch upstream for exactly this, and I bumped to it. kubevirt/csi-driver#382 added `--enable-vmi-hotplug-fallback`, merged as cb4770e4e475. With it set to false, the driver no longer calls the VMI-level `removevolume` on a VMI owned by a VM. virt-api rejects that call once `HotplugVolumes` is off. Our wrapper now passes the flag to upstream and no longer has its own hot-unplug path. `unpublishHotplugVolume` was a backport of kubevirt/csi-driver#184, and the previous pin already contained that change. Upstream detaches from the VMI when the VM is gone, then waits for the VMI status to drop the volume before it returns. So the wrapper's re-check never found the volume attached. Adding the ownership check there would have been dead code. Closes #4450 The chart leaves the flag at its default of true and does not render it. The driver image in the chart is stamped only at release preparation, so between releases main runs the previous release's binary, which does not know the flag. Go's `flag` package exits on an unknown flag. Rendering `=true` would crash-loop the CSI controller for no effect. The right value depends on the KubeVirt CR, not on the tenant, so the comment next to `HotplugVolumes` in `kubevirt-cr.yaml` points at the controller Deployment. The follow-up PR drops the gate and renders the flag as false. Nothing changes on the node side. Main already pins f5de10d39556, and cb4770e4e475 is that commit plus kubevirt/csi-driver#382. A Go unit test runs an unpublish through the wrapper during a teardown, with the flag on, off with a VM-owned VMI, and off with an orphaned VMI. It checks that the flag reaches upstream and that only upstream makes the VMI-level call. Upstream's own tests cover the ownership rule. A second test holds that an NFS volume only releases the node from its CiliumNetworkPolicy and never reaches the upstream hot-unplug. A helm-unittest pins that the chart renders no such argument. I haven't built the image or run e2e. Volume attach and detach on a tenant cluster in e2e is the real check. ### Screenshots No UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(kubernetes): kubevirt-csi supports --enable-vmi-hotplug-fallback, so it can detach volumes without the KubeVirt HotplugVolumes feature gate. The flag stays true for now and behaviour is unchanged. ``` |
||
|
|
64f43f4f77 |
fix(computeplane): do not grant tenants the cluster admin kubeconfig
The computeplane module deploys the kubernetes chart into the tenant namespace, and that chart's dashboard Role lets the tenant's "use" subjects read <release>-admin-kubeconfig. Withholding the Secret from the tenant secrets view does not help when RBAC grants it directly, so any tenant user at "use" level or above could fetch the cluster-admin credential of a cluster the module promises to keep operator-owned. The kubernetes chart now drops that rule when the internal value withholdAdminKubeconfig is set, and the module sets it in the cluster HelmRelease's inline values. The tenant cannot reach those values: the module reads only nodeGroups from them and splices an allowlisted subset into the pool releases, and Flux applies spec.values over valuesFrom. A regular kubernetes app keeps the rule, so its dashboard can still offer the kubeconfig. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
fccb7fee06 |
test(tests): fail absence-only cases when the template renders nothing
Each of these templates is wrapped in a top-level condition, and the case asserts only that a field is absent. If a values change turns the condition off, the render is empty and the absence holds trivially, so the case passes without looking at anything. A document selector makes an empty render fail with "document not found". Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
d6b38f0a79 |
fix(kubernetes): fail the OIDC bootstrap Job when the tenant CA is missing (#4232)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does In `spec.oidc.mode: System`, the post-install/post-upgrade bootstrap Job patches the chart-owned `<release>-oidc-kubeconfig` Secret with a real kubeconfig built from the tenant admin kubeconfig's CA data. When that CA data was empty, the script logged a WARNING and continued, then exited 0. The hook was satisfied, the HelmRelease reported `Ready`, and the Secret was left holding the install-time placeholder kubeconfig, which cannot authenticate. This PR makes the Job fail (`exit 1`) instead, so the failure surfaces as a red install and can be retried (the Job already has `backoffLimit: 6` and `restartPolicy: OnFailure`) rather than reported as a falsely healthy release. This is the smallest of the three directions raised on the linked issue, and matches how the neighbouring failure mode in the same script already behaves (a failed `kubectl apply` of the ClusterRoleBindings already fails the Job under `set -euo pipefail`). Behaviour for `mode: None` and `mode: CustomConfig` is unchanged, and the common case — CA data present — runs the identical patch commands as before. A regression test was added to `packages/apps/kubernetes/tests/oidc_test.yaml` asserting the rendered Job script contains the `exit 1` on empty `CA_DATA` and no longer contains the old warn-and-continue text. Confirmed it fails against the pre-fix template and passes against the fix. Validation: - `helm unittest .` in `packages/apps/kubernetes` — 150/150 tests pass (up from 149 before the new test). - `helm template ... --show-only templates/oidc-rbac-job.yaml` with representative `System`-mode values renders valid YAML; the extracted bootstrap script passes `bash -n` and `shellcheck -s sh` introduces no new warnings versus the pre-change script. - `make generate` was not run/needed — this change touches neither `values.yaml`, `values.schema.json`, `Chart.yaml`, nor `README.md`. - Not run: a live cluster reproduction (Kamaji-issued admin kubeconfig actually missing CA data at Job run time). The defect is provable from the script logic itself and the added unit test demonstrates the fail-then-pass behaviour; there is no live cluster available in this environment. ### Screenshots Not applicable — no UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: This change is confined to the shell logic inside `packages/apps/kubernetes/templates/oidc-rbac-job.yaml`'s post-install hook Job (a `Values.oidc.mode: System`-only failure path) and its accompanying helm-unittest test. It does not add, rename or remove a package, a `values.yaml` key, a CRD, a namespace, a release-owned object name, or any other surface the trigger map lists, so no downstream repository is affected. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(kubernetes): a System-mode OIDC install no longer reports Ready while the tenant OIDC kubeconfig Secret still holds the install-time placeholder — the bootstrap Job now fails (and retries) instead of only logging a warning when the tenant cluster's CA data isn't available yet. ``` Fixes #3765 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * OIDC setup now fails with an error if the tenant admin kubeconfig is missing certificate authority data, rather than leaving the OIDC kubeconfig Secret at its install-time placeholder. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
4fe45eba32 |
fix(kubernetes): fail the OIDC bootstrap Job when the tenant CA is missing
In mode=System the post-install hook Job patches the chart-owned <release>-oidc-kubeconfig Secret with a real kubeconfig built from the tenant admin kubeconfig's CA data. When that CA data was empty the script only warned and exited 0, so the hook was satisfied, the HelmRelease reported Ready, and the Secret kept the install-time placeholder kubeconfig, which cannot authenticate. The failure only showed up when a user tried to use the downloaded kubeconfig. Exit 1 instead, so the install fails visibly and is retried. The Job already has backoffLimit: 6 and restartPolicy: OnFailure, and a failed ClusterRoleBinding apply in the same script already fails the Job the same way. Assisted-by: LLM Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com> |
||
|
|
36b04218d7 |
test(kubernetes): assert where the hypervisor token is and where it is not
Every assert here was written against the mutation it exists for, and each was checked to turn red on it and green without: putting the controller back in the tenant, giving the tenant a non-empty config.clusters, pointing a sidecar at the wrong kubeconfig, restoring the blanket toleration, hardcoding the driver's default owner id, dropping capmox mode, and letting the controller ignore proxmox.csi.enabled. The negations carry `any: true`. Without it a negated containsDocument passes whether or not the document is in the render. Two of the cases are about objects rather than values: the tenant release must carry no valuesFrom at all, and two clusters must derive different volume owner ids. Both are properties a reader can check against the rendered output, which is why they are asserted that way rather than by reading the templates. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
5ab36023e0 |
fix(kubernetes): release Proxmox volumes before the driver that frees them goes
Deleting a tenant cluster left its PVC disks on the hypervisor. The pre-delete
hook deletes every HelmRelease labelled with the cluster in Step 2, the CSI
driver among them, and nothing before that deleted the tenant's PVCs — a volume
is freed only by the driver answering DeleteVolume. capmox then destroys the
worker VMs, and Proxmox frees only what the destroyed VMID owns, which these
volumes are not. They stayed on storage with the tenant's data in them, and the
PV objects that could have named their owner went with the control plane.
Step 1b deletes the claims while the driver is still up, and it deletes the
claims before the Pods that hold them: a claim being deleted is one the
scheduler places no new Pod on, so a Pod that a StatefulSet or Deployment brings
back cannot mount it again and hold it to the backstop. The Pods go second: a
claim in use carries the kubernetes.io/pvc-protection finalizer, so it clears
only once nothing mounts it, and the Pods are going with the cluster in any
case. Then it waits for the PVs rather than the PVCs, because the claim
disappears as soon as its finalizer clears while the disk is freed only when the
PV is gone.
The Pods holding a claim are found by listing every Pod with the claims it
mounts and matching in the shell. A JSONPath filter on the claim name cannot do
it: with a Pod that mounts two claims its left side yields two values, client-go
refuses the whole expression ("can only compare one element at a time"), and no
Pod in that namespace would be found for any claim.
A kubeconfig that exists is not an apiserver that answers, and listing claims
against one that does not comes back empty — the clean case. Step 1b asks
/readyz first and sends an apiserver that stays silent through the backstop, so
the summary carries it. The probe is bounded by `timeout` rather than by
--request-timeout: a non-zero --request-timeout on an in-cluster kubectl call
diverts it to localhost:8080, which is why hack/check-hook-request-timeout.bats
refuses the flag in hook templates, and the check cannot tell this call, which
carries its own kubeconfig, from one that does not.
Best effort, and loud about it: a tenant that never came up has no apiserver to
ask, and a teardown must not wedge on that, but a timeout is reported through
the same backstop the other steps use.
The hook's Role gains one Secret by name — the tenant's admin kubeconfig, which
this step needs. Without it the step takes its no-kubeconfig branch and reports
that there is nothing to release while the volumes stay behind.
Step 5 no longer fails the whole hook where CDI is not installed. It deletes
DataVolumes, which are a KubeVirt-path component, and kubectl exits non-zero on
an unknown resource type; under set -e that failed the teardown after every
earlier step had already run, leaving the cluster half removed and the hook
reporting failure.
Verified on a throwaway tenant: a PVC bound and mounted by a running Pod, then
`helm uninstall`, and the volume freed by the driver rather than by hand.
delete_hook_test.yaml pins Step 1b and the Role's admin-kubeconfig entry on
proxmox and their absence on kubevirt, the /readyz probe, and the order of the
claim and Pod deletes.
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
|
||
|
|
924d28cf3d |
feat(kubernetes): default gpu-operator to an OS-provided driver on Talos (#3585)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Closes the gpu-operator half of #3563 — gap 3 (the operator cannot validate a driver it did not install) and gap 4 (nothing routes CUDA libraries to a tenant pod). Tenant clusters have run Talos workers since 1.6.0, where the NVIDIA driver comes from a system extension baked into the worker image rather than from the operator's driver container. The addon default still described the Ubuntu shape, so a GPU node group came up with the operator fighting the OS for the driver and no path at all for the CUDA libraries. Three changes, each closing one silent failure: | Change | Without it | |---|---| | `driver.enabled: false`, `toolkit.enabled: false` | the driver container loads its own module and clashes with the extension's; the toolkit container rewrites `/etc/containerd/config.toml`, which Talos owns and which already registers the `nvidia` runtime with `enable_cdi = true` and `cdi_spec_dirs = ['/run/cdi']` | | `validator.driver.env`: `DRIVER_ROOT=/`, `NVIDIA_DRIVER_ROOT=/`, `DISABLE_DEV_CHAR_SYMLINK_CREATION=true` | the driver validator looks for a driver-container install, finds none, never writes its sentinel, and the device plugin stays blocked behind `toolkit-validation` — the loop on `Attempting to validate a driver container installation` (**gap 3**) | | `devicePlugin.env`: `DEVICE_LIST_STRATEGY=cdi-annotations`, `NVIDIA_CTK_PATH=/usr/local/bin/nvidia-ctk`, plus `DEVICE_ID_STRATEGY`, `NVIDIA_DRIVER_ROOT`, `CDI_ANNOTATION_PREFIX` | tenant pods carry no `runtimeClassName` and did not need one on Ubuntu, where the toolkit made `nvidia` the default containerd runtime. Talos does not, so CDI is the only route: the pod gets the device and no CUDA libraries (**gap 4**) | Every value is the configuration validated on a production Talos GPU node group, minus the parts that only mattered while the driver container was in play. No DaemonSet, no new image, no new package — the fix is values. ### The removed default The previous default fed `NVreg_NvLinkDisable=1` to the driver container through the `nvidia-kernel-module-params` ConfigMap. With that container off the ConfigMap is mounted by nothing, so this removes the default. On Talos the parameter belongs to the worker machine config, which the `kernelModules` field of the worker pool writes: that field is top-level in the KubernetesNodes app (`packages/apps/kubernetes-nodes`), added by #3571, and its automatic set for a pool holding an `nvidia.com/*` GPU already loads `nvidia` with `NVreg_NvLinkDisable=1`. The wrapper chart keeps the ConfigMap feature for an operator who re-enables the driver container through `valuesOverride`. A test pins the absence so the dead default cannot come back and offer a second, inert source for the same setting. ### Relationship to #3571 Independent files, no conflict, but only useful together: this makes `driver.enabled: false` correct, and #3571 is what actually loads the modules the extension ships. Merged alone, this changes a GPU node group from "operator fights the OS" to "operator stays out of the way and the modules still are not loaded" — better, but not yet working. #3571 alone leaves the operator fighting for the driver. Either order is safe; both are needed for a GPU node group to work. ### What is verified, and what is not Verified here: the rendered HelmRelease values, that `valuesOverride` still wins over each new key, and that the removed default is gone. The individual values are the ones running in production today, applied by hand through `addons.gpuOperator.valuesOverride`. Not verified here: this exact default exercised end-to-end as the chart's own output. That needs a Talos GPU node group whose modules load, which needs #3571 released. I would rather say so than imply a green e2e that does not exist — happy to hold this as draft until I have run a single-node L40S cluster against the chart default, and I will report the result on this PR either way. ### Not covered HAMi. When `addons.hami.enabled` the operator's device plugin stays off and HAMi serves the GPUs; HAMi's own plugin needs equivalent CDI wiring on Talos, which this default does not attempt. Flagging rather than guessing — the production cluster this was derived from runs HAMi disabled, so I have no validated configuration for that path. Worth its own issue if a maintainer confirms the shape. Also not in scope, and unchanged from the merge base: `talos.installerRepository` and `talos.version` reach the worker machine config unvalidated (being tracked separately by @lexfrei), and the `nvidia-operator-validator` chain for the *management* cluster's own passthrough/vgpu variants, which never load the host driver at all. ### Screenshots No UI changes. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/643 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(kubernetes): The GPU Operator addon of tenant Kubernetes clusters now defaults to a driver provided by the operating system, which is the shape of the Talos workers these clusters have run since 1.6.0. `driver` and `toolkit` are disabled (the driver comes from a Talos system extension, and containerd configuration belongs to the machine config), the driver validator is pointed at the host root so it stops looping on "Attempting to validate a driver container installation", and the device plugin is switched to CDI (`DEVICE_LIST_STRATEGY=cdi-annotations`) so tenant pods receive the CUDA libraries — previously they received the GPU device and nothing else, since Talos does not make `nvidia` the default containerd runtime. The previous default, which fed `NVreg_NvLinkDisable=1` to the operator's driver container through a ConfigMap, is removed because that container is disabled; on Talos the parameter is carried by the worker machine config, through the `kernelModules` field of the worker pool in the KubernetesNodes app (`packages/apps/kubernetes-nodes`), which loads `nvidia` with `NVreg_NvLinkDisable=1` by default on a pool holding an `nvidia.com/*` GPU. Operators who overrode `addons.gpuOperator.valuesOverride` to work around this by hand can drop those overrides. NOTE: an override that sets `devicePlugin.env` replaces the list rather than merging it, so restate the full list when overriding it or CDI stops working. HAMi is not covered: when `addons.hami.enabled` the operator's device plugin stays off and HAMi's own plugin needs equivalent CDI wiring, which this change does not attempt. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Configuration** * GPU workloads on Talos workers now use host-provided NVIDIA drivers and tooling by default; the GPU Operator’s driver and toolkit are disabled. * Device discovery uses CDI annotations and UUID-based device IDs. When HAMi is enabled, the Operator’s device plugin is disabled. * Host-root validation no longer creates `/dev/char` symlinks. * Driver kernel-module configuration is no longer created by default. * **Documentation** * Clarified when kernel-module configuration applies and what settings are needed to re-enable the driver. * Noted that `valuesOverride` replaces lists wholesale. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
669fb2416a |
feat(kubernetes): default gpu-operator to an OS-provided driver on Talos
Closes the gpu-operator half of #3563. Tenant clusters have run Talos workers since 1.6.0, where the NVIDIA driver comes from a system extension baked into the worker image rather than from the operator's driver container. The addon default still described the Ubuntu shape, so a GPU node group came up with the operator fighting the OS for the driver and no path at all for CUDA libraries to reach a tenant pod. Three things change, each fixing one silent failure: driver.enabled and toolkit.enabled go false. The driver container would load its own module and clash with the extension's; the toolkit container would rewrite /etc/containerd/config.toml, which Talos owns and which already registers the nvidia runtime with enable_cdi = true and cdi_spec_dirs = ['/run/cdi']. validator.driver.env points DRIVER_ROOT and NVIDIA_DRIVER_ROOT at / and disables /dev/char symlink creation. The driver validator otherwise looks for a driver container install, finds none, never writes its sentinel, and leaves the device plugin blocked behind toolkit-validation — the loop on "Attempting to validate a driver container installation". devicePlugin.env selects cdi-annotations and points NVIDIA_CTK_PATH at the host's nvidia-ctk. Tenant pods carry no runtimeClassName and did not need one on Ubuntu, where the toolkit made nvidia the default containerd runtime; Talos does not, so CDI is the only route for the CUDA libraries. DEVICE_ID_STRATEGY, NVIDIA_DRIVER_ROOT and CDI_ANNOTATION_PREFIX are set and tested alongside it: CDI device names key on the UUID strategy, the plugin resolves the driver from NVIDIA_DRIVER_ROOT, and the annotation prefix has to be the one containerd reads, so a regression in any of them breaks device resolution on its own. The previous default fed NVreg_NvLinkDisable=1 to the driver container through the nvidia-kernel-module-params ConfigMap. With that container off the ConfigMap is mounted by nothing, so the default is removed. On Talos a module parameter belongs to the worker machine config, which the kernelModules field of the KubernetesNodes app (packages/apps/kubernetes-nodes, #3571) writes. The wrapper chart keeps the ConfigMap for an operator who re-enables the driver container. Every value here is the configuration validated on a production Talos GPU node group, minus the parts that only mattered while the driver container was in play. Not covered: HAMi. When addons.hami.enabled the operator's device plugin stays off and HAMi serves the GPUs, and HAMi's own plugin needs equivalent CDI wiring on Talos that this default does not attempt. Refs: #3563 Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
2b7e0db40f |
fix(kubernetes): let kubevirt-csi hot-unplug from an orphaned VMI
The driver falls back to the VMI-level removevolume subresource when a volume is still attached to the VMI but no longer in the VM spec. The -kcsi Role grants only the VM-level subresources, and nothing else bound to the service account covers the VMI one: kubevirt.io: default grants no subresource of it, and the kubevirt.io:edit and admin roles that do are aggregated only into the built-in edit and admin roles, which the tenant namespace never binds to this account. Every such call ends in 403, the unpublish retries until it fails, and the hot-plug pod keeps the infra volume attached. The call stays needed once the driver stops falling back for a VM-owned VMI (kubevirt/csi-driver#382), because it still makes it for a VMI no VM owns. Deleting a VM with orphan propagation leaves its VMI running, since the KubeVirt VM controller then drops its finalizer without stopping the VMI, so the tenant node stays up and its volumes stay attached until the driver detaches them. The grant adds no reach the account lacks: it can already remove a volume from any VM in the namespace, and virt-api refuses the VMI call on a VM-owned VMI once the HotplugVolumes gate is off. virtualmachineinstances/addvolume stays ungranted, as the driver only attaches through the VM. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
06400ac033 |
fix(kubernetes): probe Keycloak CRDs by output in oidc-bootstrap cleanup
`kubectl api-resources --api-group=<absent group>` exits 0 and prints an empty list, so guarding on the exit code always fell through. The delete then failed with `the server doesn't have a resource type "keycloakclient"` — `--ignore-not-found` covers a missing object, not an unknown type. Because the oidc-bootstrap Job is a post-install/post-upgrade Helm hook, that failure blocked the Helm upgrade of every `kubernetes-<cluster>` release on any cluster without the EDP Keycloak operator CRDs, which is the default configuration (`oidc.mode: None`). Capture the discovery output once and gate each delete on its own resource type, matching the probe idiom already used by the platform migrations and the ouroboros cleanup hook. Fixes #3516 Signed-off-by: Mattia Eleuteri <mattia@hidora.io> Assisted-by: LLM |
||
|
|
8dbc7bceb5 |
feat(kubevirt)!: drop HotplugVolumes for declarative volume hotplug
KubeVirt 1.9 deprecates HotplugVolumes in favour of DeclarativeHotplugVolumes, which is Beta and therefore on by default, but virt-controller skips declarative reconciliation for as long as the old gate is on. Keeping it was only needed because kubevirt-csi detached through the VMI-level removevolume, which virt-api accepts on a VM-owned VMI only with the old gate. kubevirt-csi can now skip that call, so its fallback goes off together with the gate. The two live in different HelmReleases and roll out one after the other; in between the driver may issue a VMI-level call virt-api rejects, and the unpublish fails and is retried until both sides agree. The chart now renders --enable-vmi-hotplug-fallback=false, which a kubevirt-csi image built before the flag existed rejects at startup. The driver image is stamped at release preparation, so this needs the rebuilt image that ships in the same release. DeclarativeHotplugVolumes is now listed explicitly. With HotplugVolumes gone it is the only hotplug path, and disabling it through disabledFeatureGates would make virt-api refuse every addvolume, so the chart refuses that render instead. This changes volume hotplug platform-wide. A VM-level addvolume or removevolume now edits VM.spec directly and virt-controller reconciles the VMI hotplug volumes against it, instead of going through VM.status.volumeRequests. On the first reconcile it also removes every hotpluggable VMI volume missing from VM.spec; platform migration 59 copies those into VM.spec beforehand on the variants that run migrations. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
bb1b2a64f4 |
feat(kubernetes): tie kubevirt-csi's VMI hotplug fallback to the gate
The right value of --enable-vmi-hotplug-fallback is decided by the KubeVirt CR in packages/system/kubevirt, not by the tenant: it must stay true while that CR enables HotplugVolumes and turn false in the same change that drops the gate. A note beside the gate in the CR points at the controller Deployment, and a helm-unittest pins the flag off the rendered args. The chart relies on the driver default instead of rendering =true. The driver image is stamped only at release preparation, so between releases the chart on main runs the previous release's binary, which does not know the flag, and Go's flag package exits on an unknown one: rendering the default would crash-loop the controller for no effect. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
6602c080a1 |
fix(kubernetes): drop apex guard on oidc-rbac-job
The oidc-rbac bootstrap Job renders unconditionally, including the no-etcd bootstrap path and with OIDC disabled, so guarding its OIDC_SERVER_URL apex failed a render that must succeed: helm template of the chart with no etcd DataStore (hack/admin-kubeconfig-invariant.bats) aborted instead of emitting only the awaiting-etcd beacon. A running cluster's apiserver endpoint is already guarded in cluster.yaml (etcd present), which is the meaningful malformed-hostname case, so revert the oidc-rbac-job guard rather than gate it further. Restore the no-etcd unit fixture to an empty host so it keeps exercising that path. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
37f59369d3 |
fix(charts): fail fast on empty tenant apex in host composition
Charts that publish a service compose their hostname as <sub>.<_namespace.host>. When the injected tenant apex is empty and no per-app host override is set, they used to emit a malformed trailing-dot hostname (e.g. "harbor.") into Ingresses, HTTPRoutes and Certificate dnsNames silently. Source the apex through cozy-lib.ns-host-required at each hostname-emitting site so the render fails with a clear message instead, matching the gateway path. Covered: harbor, postgres (external), qdrant (external), kubernetes (control-plane endpoint), vpn, seaweedfs, bootbox, monitoring (grafana, alerta, oidc client), and system/bucket (adds the cozy-lib subchart symlink it lacked). The guard is placed at the point each resource is actually emitted, so standalone renders, no-exposure tenants and the kubernetes no-etcd beacon path are unaffected. The post-delete cleanup hooks are left unguarded so an empty apex cannot block uninstall. Behavior change: a tenant with an empty injected apex and no host override that enables one of these components now fails its render (HelmRelease NotReady) instead of deploying a broken hostname. In practice the apex is always injected (root has root-host, children inherit it), so this only trips a genuinely misconfigured tenant. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
4bc6621a5c |
test(kubernetes,kubernetes-nodes): assert both sides of every substrate branch
Every `if substrate` in both charts is asserted on both sides, so an inverted condition cannot pass: - the reconcile Job reads SVC_IP from the LoadBalancer status on proxmox and from the ClusterIP on kubevirt; - the machineconfig carries the pool's nameservers on proxmox and none on kubevirt; - the proxmox gate on each kubevirt-only template in the parent chart, the two RBAC bindings included, so a proxmox cluster renders no binding to a Role, ClusterRole or ServiceAccount it does not render; - the sizing conversion at 1500m and 8G, where ceil and floor disagree (2 cores and 7630 MiB against 1 and 7629); - the old-template retention loop keeps a ProxmoxMachineTemplate a MachineSet still references; - the proxmox worker monitor carries no Pod threshold while the kubevirt one does; - a live cluster and a live pool refuse a substrate switch while a first install still renders; - both charts refuse a dnsServers entry that is not an address, the parent chart included, where the value reaches ProxmoxCluster.spec.dnsServers. The negations carry `any: true`: without it a negated containsDocument passes whether or not the document is in the render. hack/talos-reconcile-heredoc_test.bats gains the dnsServers cases: the render refuses a command substitution and names the entry; an entry with a colon in it is refused too, which pins the IPv6 branch of the validation and not only the IPv4 one; and a real address survives the Job's shell unchanged. The heredoc is extracted by shape — `cat <<EOF` at the start of a line or right after `=$(` / `="$(` — so the same case keeps working whichever way the Job writes it. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
c5996c1d0f |
fix(kubernetes,kubernetes-nodes): give Proxmox workers an endpoint they can reach
A worker on the proxmox substrate is a VM on the hypervisor network, not a Pod in this cluster. It was handed two addresses only an in-cluster consumer can route to: the apiserver as a ClusterIP, and the management cluster's CoreDNS ClusterIP as its resolver. The VM booted, the Machine reported Ready with a providerID, the disk was where it should be — and no CSR ever arrived. Nothing in that picture points at an address. The control-plane chart now publishes the tenant apiserver as a LoadBalancer on the proxmox path, putting the endpoint on the network the workers share, and the pool's reconcile Job reads the Service's LoadBalancer address instead of its ClusterIP for the host entry the workers dial. The nameservers follow the same reasoning: proxmox pools carry their own resolvers rather than pointing at a CoreDNS they cannot reach, and the chart refuses to render a pool that has not named any. KubeVirt keeps the ClusterIP on both counts. Its workers are in this cluster, where a ClusterIP is the right answer and a LoadBalancer address would be an unnecessary one. Verified end to end on a real tenant cluster: the node joins, its CSR is approved, and cilium, konnectivity and coredns come up in the tenant. One operational note this surfaced, not caused by it: TalosConfigTemplate.spec is immutable, so the reconcile Job cannot rewrite an endpoint that has already been published — the template has to be deleted for the Job to recreate it. That matters only when an existing cluster's apiserver address changes. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
135eb6bac4 |
fix(kubernetes): make the Proxmox cluster object reconcilable
Two corrections found by building a real tenant cluster on Proxmox rather than by rendering the chart. Each produced a symptom that pointed away from its cause. capmox skips control-plane endpoint validation only when spec.externalManagedControlPlane is set. Kamaji owns the endpoint here, so the field was semantically true all along and merely absent — and on an absent controlPlaneEndpoint the webhook does not return a validation error, it panics (internal/webhook/proxmoxcluster_webhook.go:105). The install failed with a nil-pointer stack trace out of an admission webhook, which reads like a broken provider rather than a missing field. The cluster.x-k8s.io/managed-by annotation now renders only on the KubeVirt path. It tells Cluster API the infrastructure object is owned by something outside CAPI, so the Cluster controller never sets an ownerRef on it — and capmox's reconciler exits at "Waiting for Cluster Controller to set OwnerRef on ProxmoxCluster" without ever creating the InClusterIPPool its workers draw addresses from. Harmless on KubevirtCluster, which nothing reconciles; load-bearing on ProxmoxCluster, which capmox must. With both in place the object reaches Ready=True, the pool is created, and the Cluster reaches Provisioned. The proxmox suite's ProxmoxCluster case asserts both: the field is set and the annotation is absent. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
db839d1198 |
feat(kubernetes): add a proxmox substrate for managed clusters
Companion to the same switch in kubernetes-nodes. The control plane is Kamaji on either substrate, so this changes exactly one object — the one Cluster.infrastructureRef points at — plus the two tenant addons that exist only because the workers are KubeVirt VMs living in this cluster. A KubevirtCluster needs no spec: its workers are in the same cluster and reach the apiserver over the pod network. A ProxmoxCluster addresses an external hypervisor, so it carries what capmox cannot infer — which Proxmox nodes may host VMs, which addresses they may take, and who resolves names for them. controlPlaneEndpoint stays absent on both: the cluster.x-k8s.io/managed-by annotation hands that to Kamaji. The address pool is required rather than defaulted. capmox has no DHCP mode; it allocates from an InClusterIPPool built out of ProxmoxCluster.spec.ipv4Config, so an unset pool is not a smaller configuration but a ProxmoxCluster that never reaches Ready and worker Machines that wait on it forever. Refusing at render time turns that into a message instead of a silence. The KubeVirt cloud-controller-manager, its cloud-config and the kubevirt-csi driver are gated off on the proxmox path. They reach the workers' VirtualMachines and DataVolumes in this cluster; on Proxmox those objects do not exist, and leaving them rendered would put controllers in the tenant reconciling nothing. Their Proxmox equivalents are separate packages and land separately. The default path is unchanged: rendering it before and after this commit differs only in the per-render random bootstrap tokens, and all 19 existing suites pass untouched. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
0078727620 |
fix(kubernetes): render the OIDC UID header only from v1.33
RemoteRequestHeaderUID is alpha and off on v1.32, so the chart had to enable an alpha gate on v1.32 tenants with OIDC on to make --requestheader-uid-headers start. From v1.33 the gate is beta and on by default. No release has shipped the v1.32 path, so moving the floor to an effective v1.33 now removes a behaviour no released chart had: the chart renders no --feature-gates term of its own, and below the floor it renders no UID flag and runs none of the UID checks, as it already did on v1.31. The refusal of a list that ends in a bare flag still applies there, because it guards the --authentication-config the chart appends on every version. The refusal of a kube gate spelling mixed between bare and kube:-prefixed terms goes too. Once the chart writes no gate term, the spelling of the operator's terms no longer concerns anything it renders, and the chart already passes other gate values the apiserver refuses, unknown names and unparseable values, through to it. --emulated-version is still read, because emulation moves the effective version below the floor. Its kube version is now read from the numbers at the start of the value, with the element split at its first equals sign as toVersionMap splits it, so v1.32, 1.32.0 and 1.32-rc all read as 1.32. With OIDC on, an --emulated-version entry without quotes whose kube version has a non-zero patch component fails the render and names the entry: the apiserver refuses such a value and would otherwise only crash-loop. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
4e9f83ac0d |
fix(kubernetes)!: move or refuse extraArgs flags the control plane owns
From 26.9.4-edge the tenant control plane sets a list of kube-apiserver flags itself and drops any extraArgs entry naming one before the apiserver sees the list; 26.3.6-edge let such an entry win. An existing tenant that set, for example, its own admission plugins through extraArgs would lose them on the first reconcile after the upgrade with nothing to say so, and a flag written with its value in a separate entry leaves that value behind as a positional argument the apiserver refuses. Two of those flags have a KamajiControlPlane field the control plane renders them from. An --enable-admission-plugins=value or --kubelet-preferred-address-types=value entry is moved into that field while the field is empty, and left out of the rendered extraArgs, so the tenant keeps its setting across the upgrade without an edit. The move is computed from the values on every render; the last entry wins, as it did while the control plane kept one entry per flag name, and a repeated address type is dropped because that field is a set. The post-upgrade Job that patches extraArgs onto the KamajiControlPlane reads the same list. Every other entry naming an owned flag is refused at render time in every OIDC mode, since the chart cannot stop the drop and the refusal is where the operator learns of it. So are the two moved flags beside a non-empty field, with a value that names nothing or is quoted, and in the two-entry spelling. The list mirrors the Kamaji builder for the shape of control plane this chart renders, and the extraArgs description names the same flags. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
0f9b8dc16c |
feat(kubernetes): expose admission controllers and kubelet address types
The KamajiControlPlane carries two lists the tenant control plane renders into kube-apiserver flags: spec.admissionControllers into --enable-admission-plugins and spec.kubelet.preferredAddressTypes into --kubelet-preferred-address-types. The chart rendered neither from the tenant values; the address types were a fixed list. Pass both through. Admission controllers are rendered only when the list is non-empty, so a tenant that sets none keeps the control plane's own default set, and a list replaces that default rather than adding to it. An empty or absent list of address types renders the list the chart rendered before, so a tenant that sets nothing renders exactly as it did. The names are declared as enums copied from the KamajiControlPlane CRD at the provider pin, so the application API refuses a misspelled name where the tenant edits it instead of at apply time. Both fields are optional. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
5b370d533f |
fix(kubernetes): read every extraArgs entry for the OIDC UID flags
The tenant control plane hands the kube-apiserver every entry of a flag, so reading only the last one misses an opinion the apiserver acts on. An earlier --feature-gates entry turning RemoteRequestHeaderUID off, or an earlier --emulated-version naming kube, was previously dropped before the apiserver saw it; now it decides, and rendering the uid header flag against the later entry alone stops the control plane from starting. Repeats of --requestheader-uid-headers append, so the chart adds its own element beside a list that omits X-Remote-Uid instead of refusing to render: the union satisfies the check the refusal existed for. --feature-gates terms accumulate into one component bucket, so the chart carries only its own term rather than a copy of the operator's, and naming the kube component both bare and kube:-prefixed is refused outright, which nothing the chart renders can repair. A flag and its value written as two entries is read the same way the apiserver reads it. pflag pairs them, so the value belongs to an entry this code reads separately: the uid header flag counts as present, which is what the gate-disabled check needs, while --feature-gates and --emulated-version decide what is rendered and are not guessed at. What holds whatever such a value hides still applies next to one: an empty header element is refused, an explicit opt-out beside a header entry is refused unless the hidden value is a gate term that could undo it, and X-Remote-Uid is added beside an operator's own header list, which it can only help. The chart's other additions wait for a reading. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
e2464ea788 |
fix(kubernetes): give System-mode tenant OIDC users a UID
The AuthenticationConfiguration the chart renders mapped username and groups only, so upstream resolved every System-mode user to an empty UID and the X-Remote-Uid header the tenant apiserver publishes reached an aggregated API server carrying nothing. Map uid to the sub claim. The platform realm enables Keycloak's UPDATE_EMAIL required action, so the address the username claim carries is mutable and cannot serve as a stable identifier, while sub is the account UUID and survives a change of either. CustomConfig mode is untouched: an operator-supplied configuration is still written verbatim, uid mapping included or not. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
d870b392fa |
fix(kubernetes): render the aggregation-layer UID header for tenant OIDC
A tenant kube-apiserver publishes requestheader-uid-headers into the extension-apiserver-authentication ConfigMap only when the flag is set, and the chart never set it. Without that key the request-header authenticator on an aggregated apiserver does not know which header carries the UID and ignores it, so a caller who has a UID is served without one. Operators had to add the flag by hand through the controlPlane.apiServer.extraArgs escape hatch. This forwards a UID the caller already has and does not create one. ServiceAccount tokens carry a UID and gain from it. OIDC users authenticated through spec.oidc.mode: System do not: the AuthenticationConfiguration the chart generates maps username and groups only, and with no claimMappings.uid upstream returns an empty UID (plugin/pkg/authenticator/token/oidc/oidc.go, getUID, read at v0.35.0), so there is nothing to carry. Mapping a claim would give those users a UID and change the identity of users who have none today, which is a separate decision and is tracked separately. The half read rather than run: that an existing UID is lost without the published key comes from the controller writing the ConfigMap and the request-header authenticator consuming it, not from a live cluster. That OIDC users have no UID is unambiguous in the source named above. Render --requestheader-uid-headers=X-Remote-Uid whenever OIDC is on, plus --feature-gates=RemoteRequestHeaderUID=true on v1.32, where the gate is still alpha and the apiserver rejects the flag without it. v1.31 gets nothing: the flag does not exist there and an unknown flag is fatal to the apiserver. The deciding version is the effective one, not the image tag. --emulated-version in the same passthrough moves it down, and the gate resolves at the emulated version through featureSpecAtEmulationVersion (featureSpecAtEmulationAndMinCompatVersion by v1.35), so an emulated 1.32 needs the gate line even on a v1.35 cluster, and below 1.32 the gate is PreAlpha and cannot be enabled at all, which leaves nothing safe to render. Both flags are new, so every extraArgs shape reaching this code was rendering before it existed, and turning one of those into a render error would strand a running tenant on a HelmRelease that stops converging while its control plane carries on. The chart therefore yields to an operator entry that already satisfies the requirement, renders nothing at all when the operator turned RemoteRequestHeaderUID off, and otherwise merges its gate term into a copy of the operator's --feature-gates value. Carrying that value into the merged entry decides the gate whether the tenant control plane collapses duplicate flags into the last one or hands both to the apiserver, which merges them in the order it reads them. Operator entries are read the way the apiserver reads them: header list elements matched whole and untrimmed, folded as EqualFold folds them, and refused when one is empty after trimming, as checkForWhiteSpaceOnly refuses it; gate names compared exactly; gate values resolved as strconv.ParseBool resolves them; blanket AllAlpha and AllBeta honoured; an empty value treated as no entry at all. Each flag is judged by its last entry carrying a value. The apiserver's CSV reader strips quotes the chart does not parse, so quotes on that entry make the chart render nothing, while quotes on an earlier entry the collapse drops do not. The render fails only where a uid-headers list of the operator's describes a control plane that already refuses to start: one that omits X-Remote-Uid, one with an element that is empty after trimming, and any such list while the gate is off. Withholding the chart's own flag rescues none of them, because the flag the apiserver rejects is the operator's. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
a3df02df23 |
fix(kubernetes): name the configured realm in the OIDC denial message
The claimValidationRules message in the System-mode AuthenticationConfiguration hardcoded "in the cozy realm", so on a renamed realm it told a rejected user to look in a realm that no longer exists. Build it from the same realm the issuer uses (toString-guarded, as the derived issuer URL already is). Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
44a9a8e735 |
feat(platform): make the Keycloak realm name configurable
The realm name "cozy" was hardcoded in keycloak-configure (which owns the realm, its clients and the cozystack-cluster-admin roles), in the dashboard and linstor-gui gatekeepers, in the info kubeconfig, and in the Grafana and managed-Kubernetes apiserver OIDC issuers. Operators could not rename the realm without patching vendored templates, which `make update` reverts. The apiserver OIDC issuer is already configurable via talm oidcIssuerUrl, so pointing it at a custom realm left every SSO component on "cozy" and broke login with an issuer mismatch. authentication.oidc.realmName (default "cozy") now rides the _cluster channel so every consumer resolves the same realm, and the default keycloakInternalUrl is derived from it so the realm is named in one place. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
355630a5e7 |
fix(kubernetes): pin KubeVirt CSI sidecars to SIG Storage releases (#3934)
## What this PR does Replaces five mutable OpenShift `latest` image references (four unique sidecar images) in the KubeVirt CSI controller and tenant-node DaemonSet with versioned Kubernetes SIG Storage releases. This addresses the affected references from #3578 without claiming the other references in that issue. The selected controller versions match the sidecars already used by Cozystack's LINSTOR CSI deployment. Render tests pin every changed container name and image so a mutable tag cannot return silently. Validation: - `helm unittest packages/apps/kubernetes`: 247/247 passed. - `helm unittest packages/system/kubevirt-csi-node`: 6/6 passed. - Live MSK-prod guest-cluster canaries on 2026-08-20 and 2026-08-21 reproduced the failure with the stock chart: the `latest` sidecars exited on the old CPU with the x86-64-v3 glibc error and the controller stayed in CrashLoopBackOff. A disposable guest cluster with the pinned images reached a 7/7 Ready controller on 2026-08-19 and was removed after the check. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the four changed template/test files. No package, schema, API, default, CRD, tooling contract, or downstream hardcoded value changes. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kubernetes): pin KubeVirt CSI sidecars to versioned SIG Storage images that run on pre-x86-64-v3 CPUs ``` |
||
|
|
3daa83f4b5 |
feat(cilium): surface a kube-proxy-replacement override instead of nothing
The pin added earlier guards a cozystack developer changing the default. It cannot observe an operator override in the field, and neither can the e2e, which only ever runs the default configuration -- so the path Ivan's review actually names, addons.cilium.valuesOverride on a tenant cluster and the cozystack.networking Package's cilium component on the host, produced no signal at all. Both webhook deny policies name their webhook's backend port only, which covers a pod dialling the ClusterIP solely because Cilium performs the service DNAT ahead of its own egress hook, so an override degrades that half silently. A status-beacon ConfigMap rather than a fail(), following the -awaiting-etcd beacon in packages/apps/kubernetes/templates/cluster.yaml. Whether Cilium without kube-proxy replacement is a supported cozystack configuration is a platform decision, and refusing to render the CNI over a weakened defence-in-depth control is not a call these charts should make. The host beacon fires on a bool false, on the string "false", and on the umbrella key being deleted, because the subchart's own "false" default coalesces into the parent. The tenant beacon fires on an explicit non-true override only, since absence means the chart default applies, and it renders on the management cluster next to the HelmRelease carrying the override -- which is why the mismatch is detectable there at all rather than inside the guest cluster. Also withdraws the argument the suite header used to carry, that a target-side ingressDeny could not be re-added as a fallback. It was wrong, and the next commit adds that rule. The header now records what the two halves each buy. Makefile gets .PHONY: test for the reason this branch already argued for kubeovn-webhook, which applies with more force here now that the suite is load-bearing for two other packages: hack/package.mk's .PHONY list deliberately carries only the targets that file defines, so a path named `test` would otherwise make the target up to date and skip the recipe. Proven both ways. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
296d3008d9 |
fix(kubernetes): pin KubeVirt CSI sidecars to SIG Storage releases
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
|
69da032103 |
docs(kubernetes): point references at migration 56 after the renumber
The merge renumbered the worker-pool adoption migration from 54 to 56, but six references still named 54 (README, pre-delete-unpin hook, cluster.yaml, worker_adoption_guard test), which now points at main's redis-operator adoption migration. Repoint them at 56. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
7a9226da15 |
test(kubernetes): make control-plane-only negative asserts actually check
The 'emits no worker-pool objects' assertions used 'not: true' with 'documentIndex: -1', which scopes the check to the last rendered document only, so a stray MachineDeployment/KubevirtMachineTemplate/MachineHealthCheck/ TalosConfigTemplate anywhere else passed silently. Move 'any: true' inside the containsDocument matcher and drop documentIndex so the negation covers every document. Verified the asserts now redden when a worker object is present. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
f035da8a0f |
Merge branch 'main' into feat/kubernetes-nodes-phase2b
Resolve conflicts introduced by main: - Drop the FluxCD tenant addon (API type, deepcopy, values schema, RD) in favour of the GPU type main added; main removed the fluxcd addon wholesale. - Renumber the Phase 2 worker-pool adoption migration 54 -> 56, behind main's redis-operator (54) and fluxcd-removal (55) migrations; targetVersion 55 -> 57. - Accept the branch's removal of the in-chart talos-reconcile Job; it moved to the kubernetes-nodes chart, main only touched its comment. - Regenerate kubernetes values.schema.json / README and the kubernetes-rd ApplicationDefinition schema. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
fd8d378b73 |
fix(kubernetes): rotate bootstrap-token Job on either token half; review nits
Fold bootstrapTokenSecret into the Job content hash alongside bootstrapTokenId. talos-secrets.yaml regenerates the two token halves independently, so hashing only the id left a rotated secret no-oping a completed Job and the tenant stuck on the stale bootstrap-token. Also address review feedback: drop a review-round reference from the tracked test, correct a stale hash in a test comment, pin the alternate-image hash case exactly instead of by regex, and fix a comment indent in talos-reconcile-job.yaml. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
035bde1920 |
fix(kubernetes): correct bootstrap-token Job rationale and make it churn-free
Address review of the main-phase bootstrap-token Job. The deadlock the header claimed cannot occur on this release: the kubernetes cozyrd carries helm-install-disable-wait, which cozystack-api maps to Install.DisableWait and Upgrade.DisableWait, so there is no main-resource wait phase in front of the hooks. Rewrite the rationale around the real reason to leave the hook shape: a post-upgrade hook never runs on a release whose upgrade keeps failing, so the token Secret never lands. Fix the stale 'stays a post-install hook' note in talos-reconcile-job.yaml and the 'fails the chart upgrade loudly' comment, which is untrue once Helm no longer waits on the Job. Give the immutable Job a content-hash name suffix over its whole rendered spec, mirroring talos-reconcile: a static name would fail an upgrade that lands while the Job is live (or inside its TTL) with 'spec.template: field is immutable'. Fold the looked-up bootstrapTokenId into the hash so a rotated token also rolls the Job instead of Helm no-oping a completed one. Make the admin-kubeconfig volume optional and gate the Pod on the shared wait-for-kubeconfig init container, so it waits in init (uncounted by backoffLimit) instead of FailedMount at t0. Extend the template tests for the hashed name, its rotation on an image change, and the optional volume + init container. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
542ab899c2 |
fix(kubernetes): break bootstrap-token Job out of Helm hook to unblock fresh installs
The bootstrap-token-tenant Job was wired as a Helm post-install/post-upgrade hook. post-* hooks fire only after Helm's wait phase resolves, and the tenant kubernetes release wait blocks on MachineDeployment Ready, which blocks on nodes registering, which needs the kubelet CSR path — for which the very bootstrap-token this Job produces has to already exist in the tenant kube-system. Fresh installs and any HR reconcile that finds MD in a non-Ready state deadlock; HR.upgradeFailures climbs while post-upgrade hooks never get their turn to run. pre-install/pre-upgrade would not help either: they fire before non-hook templates apply, so <release>-admin-kubeconfig (materialised by Kamaji after KamajiControlPlane is applied) is not yet available. Move the ServiceAccount and Job to the main install phase, matching what talos-reconcile already does in this chart. Helm applies them together with the KamajiControlPlane and talos-secrets; the Job Pod stays pending until Kamaji publishes the admin kubeconfig Secret (backoffLimit covers the retry window) and then completes before MachineDeployment can converge. Helm's wait treats the Job as another resource to reach terminal state and does not gate it on MD. Extend the existing template test to pin the absence of helm.sh/hook on both the ServiceAccount and the Job, so this cannot regress silently. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |