mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
f83db514b0eb88cefc2923d17aad59daccb4e358
676
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
95dc591198 |
test(kubernetes): keep application identity out of guest log streams
Guest pods are tenant-controlled, so their lineage labels are not a trustworthy application identity, and the platform must not promote them to the stream fields the parent scopes application logs by. The guest release replaces the monitoring-agents filters and outputs wholesale, which is what keeps the parent's label lift out of guests; pin that so a later change to the guest overrides cannot bring it in. This only covers what the platform ships. A guest admin can still edit the guest's fluent-bit or write to the parent's vlinsert directly; that needs a write-path control, which the logs proposal leaves open. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
4087218a02 |
docs: move the repo-only guides onto the website (#5034)
## What this PR does Seven guides under `docs/` were operator facing and written alongside the code, and left in the source tree where no user looks. They are on the website now, so this deletes them here and repoints every reference. ``` docs/operations/backup-classes.md -> operations/services/backup-classes docs/operations/resource-presets.md -> guides/resource-management docs/storage-immutability.md -> operations/storage-class-immutability docs/operations/system-memory-limits.md -> operations/configuration/system-memory-limits docs/gateway-classes.md -> networking/gateway-classes docs/observability/distributed-tracing.md -> operations/services/monitoring/distributed-tracing docs/operations/seaweedfs-capacity-for-backups.md -> operations/services/object-storage/capacity-for-backups ``` `docs/release.md`, `docs/deprecating-applications.md` and the two SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`, a migration and a bats suite read them by path, which is the test from #3777. References point at `https://cozystack.io/docs/v1/...`. The site rewrites that short prefix to the latest version in `layouts/404.html`, so the links follow releases and pin none. Backup Classes, resource management and `storageClass` immutability exist in the latest version already. The other four pages are only in `next/` until 1.7.0 is promoted, and their links start to resolve then. The generated mongodb files come from `values.yaml` through `make generate` with cozyvalues-gen v1.7.0, the version pinned in `.github/workflows/pre-commit.yml`. Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114) and kubernetes-nodes (223), go tests for backupcontroller, presets and the operator, rd-presets-check, the migration-50 bats suite. ### Downstream repositories - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#763 Walked the rest of the trigger map against the diff. No provider follow-up: the mongodb `values.schema.json` diff is one `description` string, with no field added, removed or renamed, no enum, no default, and no `kind`, `plural` or `release.prefix` change. `packages/core/platform/values.yaml` changed in comments only and no key, so the hand written platform-package table on the site is unaffected. Please merge after cozystack/website#763, the links in the code only resolve once those pages deploy. ### Release note ```release-note docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there. ``` |
||
|
|
8d96c46994 |
fix(monitoring-agents): isolate merged log payload under log_processed
Fluent Bit's kubernetes filter runs with Merge_Log On but without Merge_Log_Key, so a parsed JSON log line is merged into the record root. An application log line can thereby forge the service fields that later pipeline stages rely on for routing and tenant isolation (kubernetes_namespace_name, kubernetes.labels, tenant, cluster, log_source). Setting Merge_Log_Key log_processed nests the parsed payload under a dedicated key so application content can no longer override top-level service fields. Applies to both the platform (system/monitoring-agents) and guest (apps/kubernetes) Fluent Bit configurations. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
44848b9c9d |
fix(kubernetes): run Proxmox CCM v0.16.1 and pin the Proxmox images by digest (#4917)
## What this PR does
`v1.7.0-rc.1` and `release-1.7` ship the Proxmox
cloud-controller-manager as
`ghcr.io/sergelogvinov/proxmox-cloud-controller-manager:v0.15.0`, a bare
tag. That tag was re-pushed upstream on 2026-09-27: it used to name the
2026-08-06 build (`sha256:fbc553921145…`) and now names
`sha256:d81702998ae1…`, a binary whose gitVersion is `0.15.0` without
the leading `v`. component-base v0.37 parses that string in an `init()`
and panics:
```
panic: version string "0.15.0" doesn't match expected regular expression: "^v(\d+\.\d+\.\d+)"
```
The CCM Deployment crash-loops, no new Node loses
`node.cloudprovider.kubernetes.io/uninitialized`, and workloads on new
workers stay Pending. On a live Proxmox cluster this went unnoticed for
five days: the existing Nodes were already initialized, and the failure
only showed when the next new worker came up. Every fresh install pulls
the broken build, so as it stands v1.7.0 would be the first stable
release with the Proxmox substrate, shipping a CCM that cannot start.
Two commits, both cherry-pick onto `release-1.7` unchanged:
1. **`fix(kubernetes)`**: run v0.16.1, pinned as tag plus digest in
`images/proxmox-cloud-controller-manager.tag`. That is the same shape
the chart already uses for the KubeVirt cloud provider. v0.16 builds
with goreleaser and stamps the version with its `v`. The config the init
container writes still parses: the cluster entry moved to
go-proxmox-pool's `ClusterConfig` with the same `url`, `insecure`,
`token_id`, `token_secret` and `region` keys, and `features.provider:
capmox` still writes `proxmox://<SMBIOS UUID>`, the providerID capmox
puts on the Machine.
2. **`fix(kubernetes, proxmox-csi)`**: pin the CSI driver from the same
registry at its current v0.20.0, with no version change. The
management-side controller goes through
`images/proxmox-csi-controller.tag`. The tenant-side node plugin goes
through `repository` + `tag` in
`packages/system/proxmox-csi/values.yaml`, which also puts it in
`hack/lib/image-refs.sh` and so in front of the release multi-arch gate.
Pinning takes the version out of the vendored chart's hands, so
`hack/proxmox-csi-driver-pin_test.bats` now fails unless the chart's
`appVersion` and both pins name the same release.
All three digests are OCI indexes for `linux/amd64` and `linux/arm64`.
The sig-storage sidecars stay on tags, as they do for the KubeVirt CSI
driver: registry.k8s.io publishes through image promotion, which does
not move a tag.
### Testing
- `helm unittest`: `packages/apps/kubernetes` 31 suites / 346 tests,
`packages/apps/kubernetes-nodes` 37 / 223, all passing.
- bats: the new test, plus `image-pin-consistency`,
`promote-rewrite-tags_test`, `overlay-main-images_test`,
`nightly-mirror_test`, `promote-retag_test`, `stitch-multiarch_test`,
`bats-strict-setup`, `bats-runner-coverage`: 107/107. The new test was
checked to fail on a node tag at another version.
- `git cherry-pick -x` of both commits onto `release-1.7` (
|
||
|
|
780b068df0 |
fix(kubernetes, proxmox-csi): pin the Proxmox CSI driver images by digest
The CSI controller in the management cluster was referenced by the bare tag ghcr.io/sergelogvinov/proxmox-csi-controller:v0.20.0, and the node plugin in the tenant by the vendored chart's default, its appVersion as a bare tag. That registry is the one whose proxmox-cloud-controller- manager:v0.15.0 was re-pushed to a build that panics, so a tag there is not a statement of what runs. Both now carry the v0.20.0 digest: the controller through images/proxmox-csi-controller.tag, like the cloud-controller-manager, and the node plugin through repository and tag in the proxmox-csi values, which also makes it visible to hack/lib/image-refs.sh and therefore to the release multi-arch gate. Both digests are indexes for linux/amd64 and linux/arm64. Pinning takes the version out of the vendored chart's hands: a chart bump would bring a new appVersion while both pins kept naming the old release, and the controller and the node plugin would drift apart from the templates and from each other without anything noticing. hack/proxmox-csi-driver-pin_test.bats fails unless the chart's appVersion and both pins name the same release. The sig-storage sidecars stay on tags, as they do for the KubeVirt CSI driver: registry.k8s.io publishes them through image promotion, which does not overwrite a tag. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
|
05a347e6f6 |
fix(kubernetes): run proxmox-cloud-controller-manager v0.16.1, pinned by digest
The Proxmox CCM was referenced by its bare v0.15.0 tag, and that tag no
longer names the image it named when the template was written. It was
re-pushed on 2026-09-27 (index sha256:d81702998ae1, replacing the
2026-08-06 build sha256:fbc553921145) to a binary whose gitVersion is
"0.15.0" without the leading v. component-base v0.37 parses that string
in the init() of its legacy metrics registry and panics:
panic: version string "0.15.0" doesn't match expected regular
expression: "^v(\d+\.\d+\.\d+)"
so the Deployment crash-loops, no Node ever loses the
node.cloudprovider.kubernetes.io/uninitialized taint, and workloads on a
new worker stay Pending. Any node that pulls the tag fresh gets it, which
is every fresh install.
v0.16 builds with goreleaser and stamps gitVersion as v{{.Version}}. The
configuration this template writes is read unchanged: the cluster entry
moved to go-proxmox-pool's ClusterConfig with the same url, insecure,
token_id, token_secret and region keys, and features.provider: capmox
still writes proxmox://<SMBIOS UUID>, the providerID capmox puts on the
Machine.
The reference now lives in images/proxmox-cloud-controller-manager.tag
as tag plus digest, the shape the chart already uses for the KubeVirt
cloud provider, so a moving tag can no longer change what runs.
Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
|
||
|
|
ae9e7c4c64 |
test(kubernetes): pin RetryOnFailure on the Proxmox csi release (#4969)
The KubeVirt csi and cilium releases carry a helm-unittest pin on `strategy.name: RetryOnFailure` for both actions, and the e2e HelmRelease guard in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` names those two tests as load-bearing. `helmreleases/csi-proxmox.yaml` sets the same strategy for the same reason and had no pin. This adds `packages/apps/kubernetes/tests/csi_proxmox_install_retry_test.yaml`, a mirror of `csi_install_retry_test.yaml` rendered on the proxmox substrate, and updates the guard comment that pointed at the gap. No template change. Verified with `helm unittest .` in `packages/apps/kubernetes` (32 suites, 349 tests). Closes #4731 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Expanded automated coverage for Proxmox CSI installation and upgrade retry behavior, including the retry interval and remediation retry limits. The checks verify that retries continue according to the configured settings during installation and upgrades. This update does not change user-facing functionality. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
b43196e549 |
fix(kubernetes): pin the Proxmox cloud-config builders to the bundled image
The Proxmox substrate added two init containers that build the cloud config from the hypervisor token Secret, and both still took their image from images.waitForKubeconfig. With that key gone the render failed on a nil map; with it set, a tenant chose the image that reads the token. They now run the bundled busybox through cozy-lib.image, like the wait-for-kubeconfig container next to them. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
1636349a63 |
fix(api): point the removed image overrides at node registry mirrors
The warning for a removed image override told an air-gapped install to use the platform-wide registry, but nothing in the platform writes _cluster.images-registry, so there is nothing to set. What serves these images offline is what serves every other platform image: the registry mirrors configured on the nodes, as the air-gapped install guide sets up. The tests that hold the cozy-lib.image routing now say what they check rather than promising a platform knob. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
dfff5c2dac |
test(kubernetes): hold the routing of the busybox and unpin images
The busybox helper shared by four workloads and the pool's pre-delete Job had no test holding their route through cozy-lib.image, so reverting either line left both suites green. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
d99a1b9f2f |
fix(kubernetes): route the helper images through cozy-lib.image
These images now go through cozy-lib.image like the harbor, kafka and mariadb hook images, so they follow _cluster.images-registry wherever that is set rather than any per-application value. An air-gapped install serves them the way it serves every other platform image, through the registry mirrors configured on its nodes. The talos-csr-signer sidecar and the OIDC Jobs had no test holding their image, which is where an override could come back unnoticed. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
af20694359 |
fix(kubernetes)!: drop the helper image overrides
images.waitForKubeconfig, images.kubectl and images.talosCsrSigner ran whatever image a tenant named inside the management cluster: the csr signer as a sidecar of the tenant control plane with the Talos CA key mounted, the other two in pods holding service account tokens for CAPI, KubeVirt and Cilium objects in the tenant namespace. A tenant never gets that access through the apps API, and the platform should not hand it over through an image field. The images now always come from the bundled tag files. A release that still carries the keys renders exactly as one without them. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
b83cc087cc |
docs: move the repo-only guides onto the website
Seven guides in docs/ were written next to the code and left in the source tree, where no operator looks. The website carries them now, so a second copy here only lets the two drift again: the Backup Classes page had already fallen behind its original. docs/release.md, docs/deprecating-applications.md and the two SeaweedFS runbooks in docs/operations stay. A migration, a bats suite, a workflow or AGENTS.md reads each of them by path, which is the test for keeping a document in this repository. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
98d0402096 |
test(kubernetes): pin RetryOnFailure on the Proxmox csi release
The KubeVirt csi and cilium releases carry a helm-unittest pin on strategy.name: RetryOnFailure for both actions, and the e2e HelmRelease guard names those two tests as load-bearing. helmreleases/csi-proxmox.yaml sets the same strategy for the same reason and had no pin. Add the mirror test and update the guard comment that pointed at the gap. No template change. Closes #4731 Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
f3486f4e3a |
docs: move the operator guides out of docs/ to where readers find them
Five guides in docs/ were written next to the feature code that needed them and left in the source tree, where no operator looks. They are on the website, so a second copy here only drifts: edits kept landing in docs/ after the website copy was made. docs/vm-external-vlan.md is split rather than moved. Its section on staging the reference CNI plugins is a contract that hack/cni-plugins-staging-contract.bats asserts against the multus Dockerfile, the daemonset manifest and the platform values, so that half is package documentation and goes to packages/system/multus/README.md, which the suite now reads. The operator half goes to the website. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
953b965e66 |
fix(kubernetes): keep NFS publish from hanging on a dead server (#4791)
## What this PR does When the kubevirt-csi node plugin publishes an NFS volume, it first mounts the export root into an `nfs-init-*` temp dir to create `/data` and move old files into it. That temp mount was hard. If the NFS server stopped answering at that point, the publish call never returned. kubelet gives up after 2m and retries, each retry blocked the same way, and goroutines and temp dirs piled up until the server came back or the plugin restarted. The fix does three things: 1. The temp root is now mounted `soft` with `retry=0`. It only does metadata work that a retried publish redoes, so an error there is harmless, and a stuck call now ends on its own once the connection to the server is declared dead. The mount the pod gets stays hard, because a soft mount can report a failed write that the server did apply. 2. A publish to a target path that already has one running now returns `Aborted` instead of starting another. This is what stops the pile-up: at most one call per target path is stuck at a time. 3. Moving old files into `/data` now checks the destination with `lstat` and fails the publish on any error other than a missing file. With the soft mount, the check can fail with EIO on a file that `/data` already holds, and a rename issued after the server comes back would replace that file. `lstat` keeps a user's symlink in `/data` that loops or dangles from failing every publish. Such a symlink counts as present and is left alone, while before this change the rename replaced a dangling one. I kept the default retry timeouts on purpose, although the issue suggested lowering `timeo` and `retrans`. The Linux client shares one connection per server for all of that server's mounts on the node, and the first mount sets the connection's TCP timeouts. Lowering them on the temp mount would have changed the timeouts of the pods' hard mounts too. Over NFSv4, a soft RPC fails only after the connection is declared dead, so a stuck call takes minutes to end, longer than kubelet's 2m deadline. The guard keeps kubelet's retries from adding new calls during that time. The guard works per target path, not per volume. Two pods on one node that use the same RWX volume still publish in parallel, and each pod can have at most one stuck call. The unit tests pin the mount options on both mounts, the guard behaviour including its release after a failed publish, and the destination check: a failed check stops the move, and a symlink in `/data` stays in place. They use the fake mounter. I did not test against a real NFS server that went away. Not covered here, and tracked in #4814: - If the server's TCP connection stays up while the NFS service hangs, a soft NFSv4 RPC does not time out. The guard still limits it to one stuck call per target. - `Mount` from mount-utils runs `mount` with no deadline. This predates this change, and the guard bounds it the same way. - If the target is already hard-mounted from the dead server, the check of the target at the start of a publish blocks for good. The guard keeps this to one call per target but does not end it. Unpublish is not coordinated with a publish that is still running either. Both predate this change. - A follow-up comment on the issue reported two leftovers of the bounded unmount: an orphaned `umount.nfs` after a timeout, and a possible `stat` before `umount2`. Both may end sooner now that the temp mount is soft, but I have not verified either. Refs #4581 ### Screenshots Not a UI change. ### Downstream repositories The change is limited to the kubevirt-csi node plugin's Go code. None of the triggers in the contributing guide's map apply: no package added or renamed, no platform values, variants, release assets, ApplicationDefinition semantics or developer tooling touched. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(kubernetes): an NFS volume publish in tenant Kubernetes clusters no longer hangs forever when the NFS server goes away during the publish, and kubelet retries no longer pile up while a publish is stuck. The plugin's temporary mount of the export root is now soft, and a second publish to the same target path is refused while the first is still running. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Concurrent NFS publish requests for the same target now return an `Aborted` error. Different targets remain available, and a target can be used again after a request finishes, including after a failed attempt. * Migration now stops if checking a destination encounters an unexpected error. Existing destinations, including symlinks, are left in place. * Temporary-directory creation and mount errors now identify the affected volume. * **Updates** * Temporary NFS root mounts now use soft behavior with retries disabled. The final data mount remains hard. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
904bc38fc3 |
fix(kubernetes): delete the Talos PKI and oidc-bootstrap Job on uninstall
cert-manager runs with enableCertificateOwnerRef: false, so the <release>-talos-ca and <release>-talos-tls-cert Secrets carry no ownerReference to their Certificates and outlive the release, keeping the CA private key of a cluster that no longer exists. A new cluster created under the same name then adopts that stale CA instead of issuing its own trust root. The <release>-oidc-bootstrap Job is a hook whose only delete policy is before-hook-creation. Hooks are not part of the release manifest, so uninstall never removes it. The pre-delete hook deletes the two Certificates before their Secrets, because deleting a Secret while its Certificate exists makes cert-manager re-issue it. It runs after the TenantControlPlane is gone, since the control plane's CSR signer mounts the CA. The Role grants each delete by resourceName only. Assisted-by: LLM Signed-off-by: Ray Low <crossrt@gmail.com> |
||
|
|
bcde187858 |
fix(dashboard): small Console fixes from the #3828 triage (#3837)
## What this PR does Six small Console fixes from the #3828 triage. Each one is small and most of the code sits in `apps/console/src`, so they go together in one PR. `Fixes #3105`: sizes between 1Ki and 1Mi printed as a raw byte count. `humanizeBytes` had branches for Ti, Gi and Mi and then fell through to bytes. It now has a Ki branch with the same `toFixed(0)` as the Mi branch. The `1023B` case still holds, and the test covers 1Ki and 512Ki. `Fixes #3106`: the tenant picker showed on admin pages where the tenant changes nothing. It is now hidden on the cluster-scoped admin pages only: Capacity, Backup classes, External IPs and Modules. The rest of `/admin` mounts the same tenant-scoped pages as `/console`, and those take their namespace from the tenant context, so the picker stays there. `/admin/tenants` keeps it too, because its rows come from the picker-filtered tenant list. `Fixes #3107`: both capacity drill-downs showed one generic error, so a permission failure looked like a broken page. Both now use the `error instanceof K8sApiError && error.status === 403` check that `ClusterStorageSection` already uses, with a test for the forbidden case and one for a generic error on each page. `Fixes #3102`: an immutable leaf was not restored when the YAML edit dropped its parent object. `overlayPath` stopped as soon as the submitted body had nothing at a path segment. It now creates `{}` for the missing parent, but only when the target there is undefined or null and the original really has the leaf to restore. So a scalar or an explicit null the user put there survives. A wildcard with more path under it does not rebuild a deleted parent, since that would put back entries the user removed. The test uses foundationdb's `storage.storageClass`, which is a shipped path of this shape. `Fixes #3135`: a blocked submit did not scroll to the field that blocked it. Plain `focusOnFirstError` crashes here, because RJSF's built-in handler reads `form.elements` and this form is `tagName="div"`. So the form passes a small handler that finds the field by its generated id and scrolls it into view. The ids come from one shared definition, and the widgets that had none now carry it: the `DynamicOptionsWidget` select, which every `x-cozystack-options` field uses, and the `SourceField` controls. The nested form that `AdditionalPropertiesField` mounts per map entry continues the outer id namespace instead of restarting at `root`, so fields like `nodeGroups.*.instanceType` resolve. A map key with a dot in it, such as a mariadb user `john.doe`, resolves by trying both readings of the dotted error path. `Fixes #3108`: `cilium`, `coredns` and `verticalPodAutoscaler` have only `valuesOverride` and no `enabled`, so they rendered as plain groups next to the toggleable addons and looked like a switch that failed to appear. Setting `<addon>.enabled: true` in the YAML editor is accepted, because the schema sets no `additionalProperties`, and the API stores the field and echoes it back. Nothing reads it. The three addons are not the same case. `cilium` and `coredns` are always installed. `verticalPodAutoscaler` has no switch of its own and is installed and removed together with `addons.monitoringAgents.enabled`. The three are now marked in `values.yaml` with a new `x-cozystack-no-enable-switch` schema keyword, and the form renders them with a note that there is no enable switch and an `enabled` field in YAML does nothing. The per-addon reason goes into the schema description, which the form renders and which regenerates into the README, the Go types and the ApplicationDefinition. The generated files are regenerated with cozyvalues-gen v1.7.0, the version CI pins, and the pre-commit check passes. The console typecheck and tests pass in CI. ### Screenshots ##### Ki formatting <img width="1182" height="184" alt="image" src="https://github.com/user-attachments/assets/62662265-4f77-455d-b90e-85e417684bb9" /> ##### Admin without tenant picker <img width="1440" height="245" alt="image" src="https://github.com/user-attachments/assets/99aff4f3-fda9-4c4c-8cf8-0c1eb01fb973" /> ##### Capacity 403 error <img width="1232" height="270" alt="image" src="https://github.com/user-attachments/assets/f4e4b328-ee5e-4516-b78f-9b5084b6bbc8" /> <img width="1232" height="944" alt="image" src="https://github.com/user-attachments/assets/142e438e-f538-436e-aae8-dfb79ec747e8" /> ##### Validation focus <img width="1166" height="135" alt="image" src="https://github.com/user-attachments/assets/89abb73b-a624-406f-9736-3bf38bb8a923" /> ##### Addon lifecycle <img width="1166" height="591" alt="image" src="https://github.com/user-attachments/assets/14a95891-84d1-4073-a009-bc0eab94f114" /> ### Downstream repositories I walked the trigger map in `docs/agents/contributing.md` against the diff, file by file. This PR changes field descriptions, adds the `x-cozystack-no-enable-switch` schema keyword on three addon objects, and changes the console. No field is added, removed or renamed, and no default, version enum, `kind`, `plural`, `release.prefix`, output Secret or Service changes, so none of the terraform provider triggers fire. No package is added or removed, and nothing changes in `packages/core/platform/values.yaml`, the variants, `hack/` or the node prerequisites. cozyvalues-gen itself is unchanged, since the keyword goes through its generic `@x-*` passthrough. The kubernetes reference page on the website regenerates from `README.md` on a stable tag, which is automated rather than a follow-up. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(dashboard): Ki-range sizes now render as Ki instead of raw bytes, the tenant picker is hidden on cluster-scoped admin pages, capacity drill-downs distinguish a permission error from a broken page, an immutable field whose parent object is absent is now applied, a blocked submit scrolls to the field that blocked it including fields inside maps and the source and dropdown widgets, and the cluster addons with no enable switch now say so instead of showing a toggle that never appears ``` |
||
|
|
f76dedf9ca |
fix(kubernetes): keep NFS publish from hanging on a dead server
The kubevirt-csi node plugin mounts the NFS export root into a temp dir to create /data and migrate old files. That mount was hard, so with the server gone the mkdir, readdir and rename calls on it never returned. kubelet gives up after 2m and retries, and every retry blocked the same way, piling up goroutines and nfs-init-* dirs. Mount the temp root soft, so its RPCs fail once the connection to the server is declared dead instead of blocking forever; it only does metadata work that a retried publish redoes. retry=0 keeps mount.nfs from retrying a failed temp mount for 2m. timeo and retrans keep their defaults, because the first mount of a server sets them for the connection the pods' mounts of it share. The volume mount the pod gets stays hard, since a soft mount can report a failed write the server applied. Over NFSv4 that soft failure takes minutes, and a stat of a target that is already hard-mounted can block for good, so a publish for a target path that already has one running now returns Aborted instead of starting another. A soft stat can fail with EIO on a file /data already holds, and a rename issued once the server is back would replace it, so the migration now fails the publish on any stat error other than a missing destination. It checks the destination with lstat, so a user's symlink in /data that loops or dangles counts as present instead of failing every publish or being replaced by the rename. The skip it makes for a destination already in /data leaves the root entry where the pod cannot see it, so it is now logged as a warning. /data is created with a plain mkdir, since with the server gone MkdirAll fails its stat on the mount point and then reports the local temp dir as already existing instead of the EIO. The temp mount errors now name the volume, like the migration errors do. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
5486d4ffcf |
fix(networking): harden webhook policy assumptions (#4058)
## What this PR does This is follow-up for [#3997](https://github.com/cozystack/cozystack/pull/3997). It reads ingress-nginx admission port from effective subchart values and refuses empty value, and fixes `kubeProxyReplacement=false` status because cozystack has no supported Service translator in that state. Gateway test now requires one real ClusterIP and keeps same ready Pod UID, IP and port for whole probe, so stale EndpointSlice, replaced Pod or wrong Service can't report deny. Policy selectors and two-ended deny from [#3997](https://github.com/cozystack/cozystack/pull/3997) stay unchanged, and this does not claim root cause for [#3983](https://github.com/cozystack/cozystack/issues/3983). ### Screenshots Not applicable. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(networking): use effective ingress admission port, report unsupported Cilium service routing correctly, and reject stale backend evidence in gateway tests ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Admission webhook connectivity checks now validate the configured Service and its exact backend endpoint throughout probing, improving failure detection. - Admission webhook policies now fail closed when the webhook port is not explicitly configured. - **Documentation** - Clarified that supported service routing requires `kubeProxyReplacement: true`. - Updated guidance to explain that disabling this setting or webhook policies does not provide a supported fallback. - **Tests** - Added coverage for missing webhook port configuration and strengthened networking stability checks. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
ac4ab1c4a6 |
chore: drop private cluster and host names from the tree (#4461)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does I remove internal cluster names, a real hostname with its IP addresses, internal mirror hostnames and an internal product name from comments, an example file, test fixtures and a dead commented-out config block. Where a stand-in is needed I use neutral test-cluster wording, RFC 2606 domains or RFC 5737 documentation addresses. I also replace two real Keycloak usernames and a reviewer's name in code comments with placeholders, or drop them. Rendered output does not change. The test fixture hostnames change on the input and the expected side together, and the unit tests of every touched package still pass. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Updated backup, OIDC, autoscaling, and platform references to describe test environments without naming specific development clusters. * Replaced development-specific endpoints in the platform example with `example.org` and documentation-only IP addresses. * Updated OIDC username examples. * **Tests** * Updated fixture hostnames and expected URLs to use `example.org`. * **Chores** * Removed commented-out registry mirror entries; the active mirror configuration is unchanged. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
9a59f1d0f7 |
fix(dashboard, kubernetes): declare the no-enable-switch addons and harden the focus helper
Use the shared RJSF ID configuration in the focus helper, preserve quoted dotted property names, and restrict its fallback to form inputs. Add route and scalar-ancestor regression coverage for the rebased admin hierarchy and immutable overlay. Mark Kubernetes addons without their own switch explicitly in the generated schema instead of inferring the behavior from their current fields, and keep lifecycle-specific copy in each field description. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
35abe2dcbd |
fix(dashboard, kubernetes): say which cluster addons have no enable switch
cilium, coredns and verticalPodAutoscaler carry only valuesOverride and no enabled, and the addon template keys its toggle off the presence of enabled, so all three rendered as plain groups among the toggleable addons, reading as a switch that failed to appear. That is what leads users to add <addon>.enabled: true in the YAML editor, where the schema sets no additionalProperties, so the API stores the field and echoes it back while nothing reads it. The three are not the same case, so the form must not claim they are. cilium and coredns are always installed. verticalPodAutoscaler has no switch of its own but is installed and removed together with addons.monitoringAgents.enabled, so calling it mandatory would be wrong. The fixed copy states only what holds for all three, that there is no enable switch and that an enabled field in YAML does nothing, and the per-addon reason moves into the schema description, which the form already renders and which regenerates into the README, the Go types and the ApplicationDefinition. Also types the test fixture instead of spreading it as object, which erased the props and left the file failing tsc. Fixes #3108 Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
c9558f5658 |
fix(networking): correct webhook policy assumptions
Read the ingress admission port from the effective subchart values and fail if the port is absent. Describe kube-proxy replacement as a required service-routing invariant while retaining target-side enforcement. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
92bf90cc7d |
fix(kubernetes): force the kubevirt-csi temp NFS unmount after a timeout
The NFS publish path temp-mounts the export root and unmounts it in a defer with a plain umount, which has no deadline. If the NFS server is gone by then, that umount can block indefinitely: the publish hangs and the temp mount stays behind for the life of the node plugin container. mount-utils' UnmountWithForce cannot fix this. It retries with -f only after its first umount returns, and that umount forks the umount.nfs helper: killed on the timeout, umount leaves the helper stuck on the vanished server with the output pipe still open, and CombinedOutput waits for the pipe, so -f never runs. The plugin therefore runs umount itself with WaitDelay set, which closes the pipe shortly after the kill, and falls back to umount -f under a timeout of its own. As in mount-utils, "not mounted" counts as done: umount2 detaches the mount before it blocks tearing the superblock down, so the -f after a killed umount can find nothing left. A mounter that is not the exec-based one keeps its plain Unmount. An umount in uninterruptible sleep outlives any kill, so the cleanup also waits at most one fixed budget, well inside the 2m deadline kubelet puts on NodePublishVolume, and leaves a stalled unmount running in the background, where it logs its result once it ends. The temp dir is removed by that background unmount once it succeeds, not by the publish: removed at the deadline it is still a mount point, and nothing would retry. A forced unmount runs nfs_umount_begin, which kills every RPC in flight on the superblock, and NFS shares one superblock between mounts of the same export with the same options. The temp mount therefore gets nosharecache, so -f cannot fail the I/O of pods on the node that mount the same volume. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
520af57905 |
chore: drop private cluster, host and personal names from the tree
Private infrastructure names, a real hostname, a commented-out internal mirror block, an internal product name, and a reviewer's name in a comment do not belong in a public repository; drop them and replace what needs a stand-in with neutral placeholders. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
d2e3eb4f82 |
fix(kubernetes, e2e): unblock autoscaling below min size, size the computeplane pool, fix the backupcontroller build (#4768)
## What this PR does Makes the backupcontroller test package compile again on main. #4587 added a flavor argument to `createCNPGBackupArtifact` and `marshalCNPGBackupSnapshot`, and #4669, whose CI ran before that, added two test calls with the old argument lists. Both PRs were green on their own, and together they break `go test ./internal/backupcontroller/`. The two calls now pass the same default flavor as the other tests in the file. The second commit fixes an e2e suite that fails the required E2E check here. It is the `computeplane` suite from #4713, and its worker pool was one node with a single vCPU. After the kubelet reservations that node has 900m of CPU to allocate. The tenant addons (cilium, coredns, csi, ingress-nginx, metrics-server, cert-manager) request exactly 900m between them. If cert-manager's startupapicheck hook pod is created last, it can't be scheduled and the install times out after 10m. The reinstall after remediation runs into the same full node. Whether that pod comes last depends on install order, so the suite passed on #4713 and failed here. The pool now has two workers with one vCPU each, the same shape the kubernetes suites use. One vCPU per worker stays on purpose, because of #3513. Two workers exposed a deadlock in the tenant cluster-autoscaler. The suite's pool is created at zero replicas and then gets a min size of 2, which CAPI does not apply to the live replica count on upgrade (#4786). The autoscaler's scale-up for a pending pod asks for one node, and the clusterapi provider refuses it because one is under the min size, although it is an increase. That refusal backs the group off, and the min-size enforcement skips groups in backoff, so the pool never leaves zero. Upstream had the same report in kubernetes/autoscaler#4942, closed as stale, and it is now tracked in kubernetes/autoscaler#10400. Our autoscaler image already patches the mirror case for max size, and the patch now refuses only a decrease below the min size. `go vet` and `go test ./internal/backupcontroller/...` pass, and so do the bats pins for the computeplane suite. The autoscaler patch adds a test for each direction, and the clusterapi package tests pass with it applied to 1.32.0. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved cluster autoscaling for worker groups outside configured size limits: groups below their minimum can scale up toward it, and nodes can be removed when a group exceeds its maximum. Scale-down requests below the minimum remain restricted. * **Tests** * Updated backup and restore test calls with PostgreSQL and empty flavor values. * Updated ComputePlane checks for a two-worker pool and clarified cert-manager installation timing expectations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
82ae9f6cf7 |
fix(kubernetes): let the autoscaler grow a node group that sits below min
The clusterapi provider refuses any SetSize under the group's min size, including an increase. A pool whose min size is raised while it stands at fewer replicas then deadlocks: the scale-up for pending pods asks for fewer nodes than the min, is refused, and puts the group in backoff, and the min-size enforcement skips groups in backoff. The group stays where it is for good. Refuse only a decrease below min size. The guard it keeps is the one DecreaseTargetSize relies on, and DeleteNodes checks min size itself. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
bcda5c893a |
feat(kubevirt)!: drop HotplugVolumes for declarative volume hotplug (#4453)
## What this PR does This turns off the deprecated KubeVirt `HotplugVolumes` gate and moves the platform to declarative volume hotplug. It changes how every volume hotplug on the platform works, so the title carries `!`. Closes #4032. This PR is stacked on #4451 and has to merge after it. #4451 adds the kubevirt-csi flag this PR flips. The replacement gate is already on in KubeVirt 1.9, and the old gate is the only thing keeping it off. `DeclarativeHotplugVolumes` is Beta at v1.9.0, and Beta gates are on by default. virt-controller skips `handleDeclarativeVolumeHotplug` while `HotplugVolumes` is on, so dropping the old gate is what switches the new one on. The kubevirt-csi fallback goes off in the same change. With the old gate off, virt-api rejects the VMI-level `removevolume` on a VM-owned VMI, so the chart now passes `--enable-vmi-hotplug-fallback=false`. The comments next to the gate and the flag point at each other. This needs the rebuilt kubevirt-csi image. A driver built before #4451 does not know the flag and exits on it at startup. The image is stamped at release preparation and ships in the same release as this change, so only a `make apply` from main before then would crash-loop the CSI controller. What changes at runtime on an upgrade: - The VM-level `addvolume` and `removevolume` calls now edit `VM.spec` directly. virt-controller then makes the VMI hotplug volumes match it, instead of going through `VM.status.volumeRequests`. - Volumes hot-plugged through the VM subresource are in `VM.spec`, so they stay attached. - During a tenant teardown, when the VM goes before its VMI, the driver no longer calls the VMI subresource. It waits for the VMI status to drop the volume, and the garbage collector deletes the VMI with its VM. Upstream waits up to two minutes. If the VMI is still there after that, the unpublish fails and is retried. A new platform migration keeps old VMI-only volumes attached. On its first pass virt-controller removes every hotpluggable VMI volume that is missing from `VM.spec`. Until July 2025 kubevirt-csi attached volumes at the VMI level (the Role then granted `virtualmachineinstances/addvolume`), and live migration carries such a VMI across KubeVirt updates. Without help, a tenant node not recreated since then would lose a mounted disk on upgrade. Migration 59 copies each such volume and its disk from the VMI into the VM template before anything else changes. Adding a hotplug volume to the VM spec is a valid live update in KubeVirt, so no VM needs a restart. It runs as the platform chart's pre-upgrade hook, before the upgrade re-points `cozystack-packages`. The KubeVirt CR and the tenant kubernetes charts both render from that source, so the gate and the flag cannot change before the migration has finished. It only touches VMs that need it, logs each one, and a second run changes nothing. The migration also makes ephemeral hotplugs permanent. A volume a user added at the VMI level, for example with `virtctl addvolume` without `--persist` from an older virtctl, has the same shape as kubevirt-csi's old attachments, because virt-api builds both the same way. There is nothing reliable to tell them apart by, so both are copied. Leaving the user's volume out would not keep it ephemeral: the first reconcile would unplug it from the running VM. After the upgrade it is part of `VM.spec` and attaches again on a restart, and `virtctl removevolume` drops it. Ephemeral hotplug stops working on VMs. With the old gate off, virt-api refuses VMI-level `addvolume` and `removevolume` on a VMI that a VM owns (`vmiVolumePatch` in v1.9.0). A volume on such a VMI is added and removed through the VM from then on. A VMI without a VM is not affected. Only running VMIs are copied. virt-controller reconciles no other VMI, and a stopped VM builds its next VMI from `VM.spec`, so copying from a stopped one would bring back volumes the restart was going to drop. Migrations run only on the isp-full, isp-full-generic and isp-hosted variants. KubeVirt comes from the platform only on isp-full and isp-full-generic, so those are covered. On the default variant, where packages are installed by hand, nothing runs the migration. This optional check lists the VMIs and volumes the migration would copy. It uses the same rule. On the default variant, run it before upgrading and recreate each tenant node it lists: ```bash kubectl get vm,vmi --all-namespaces --output json | jq --raw-output ' ([.items[] | select(.kind == "VirtualMachine") | {key: "\(.metadata.namespace)/\(.metadata.name)", value: [.spec.template.spec.volumes[]?.name]}] | from_entries) as $vm | .items[] | select(.kind == "VirtualMachineInstance") | ([.metadata.ownerReferences[]? | select(.controller == true and .kind == "VirtualMachine" and (.apiVersion | startswith("kubevirt.io/"))) | .name][0]) as $owner | "\(.metadata.namespace)/\($owner)" as $key | select(.status.phase == "Running" and $owner != null and $vm[$key] != null) | .spec.volumes[]? | select(.dataVolume.hotpluggable == true or .persistentVolumeClaim.hotpluggable == true) | .name as $n | select($vm[$key] | index($n) | not) | "\($key) \($n)"' ``` Empty output means nothing is affected. I checked the query against the migration's test fixtures only. Neither the query nor the migration has run on a live cluster. #4287 also adds a migration 59. If it merges first, this one moves to 60: the file, the version it stamps, `migrations.targetVersion`, the bats file (its `run_migration` argument and expected `STAMP`) and `hack/testdata/migration-59-vmi-hotplug`. A VM with a pending hotplug request at upgrade time may also get stuck. virt-controller still processes `VM.status.volumeRequests` before the declarative handler, and it applies them through the VMI subresource, which virt-api rejects once the old gate is off. The sync then fails before the VM spec is updated, so nothing clears the request, and every later sync of that VM fails the same way. A retry through the VM subresource edits the spec and clears it, which covers kubevirt-csi's own requests. A one-shot request stays stuck until the VM restarts. I read this from the v1.9.0 code and haven't reproduced it. Waiting until this lists nothing before upgrading narrows the window, but a request made during the rollout can still hit it: ```bash kubectl get vm --all-namespaces --output json | jq --raw-output '.items[] | select((.status.volumeRequests // []) | length > 0) | .metadata.namespace + "/" + .metadata.name' ``` The two halves live in different HelmReleases, so there is a short window during the rollout where the driver and the gate disagree. In that window an unpublish fails and is retried until both are updated. A tenant cluster whose `kubernetes` HelmRelease is suspended or failing keeps its old driver for longer. Drivers from before July 2025 attach through the VMI subresource, so on such a tenant every attach is refused until its release reconciles. Such a driver can also attach a volume after the migration has run and before the gate flips. The migration doesn't see that volume, and the first declarative reconcile unplugs it from the running VM. The migration lists the VMs once and patches each one with a `resourceVersion` test, so a VM that changes in between fails the hook Job. The Job has `backoffLimit: 3`, and each retry lists again and skips the VMs already patched. Re-reading only the conflicting VM would avoid the retry; I left that out. The new gate is listed in the CR even though it is on by default. With the old gate gone it is the only hotplug path, and the chart refuses a render that disables it. I found no other user of `HotplugVolumes` in the platform or in the downstream repositories. The VMI-level fallback may have been failing already. The kcsi Role has no rule for the VMI-level `removevolume` (#4452), so that call likely got 403 with or without the gate. I haven't checked that on a cluster. `hack/migration-59-vmi-hotplug-volumes.bats` runs the migration against a fake kubectl: a VM that needs the copy, VMs and VMIs that must be left alone (a VMI that is not Running among them), a second run, a cluster without KubeVirt and the fail-closed paths. The helm-unittests for the KubeVirt CR pin the whole gate list, accept `HotplugVolumes` in `disabledFeatureGates` and refuse `DeclarativeHotplugVolumes` there. The CSI test pins the flag at false, and the `make update` idempotency test still passes. None of that exercises hotplug itself. The real test is e2e with volume attach and detach on a tenant cluster, including a teardown. No CI lane upgrades from the previous release yet, so an upgrade with a live VMI-only volume is untested; #4463 tracks that test. ### Screenshots No UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(kubevirt)!: KubeVirt now runs with DeclarativeHotplugVolumes instead of the deprecated HotplugVolumes feature gate, and kubevirt-csi no longer uses the VMI-level hot-unplug. Volume hotplug requests are applied through VM.spec. The upgrade copies volumes that older kubevirt-csi versions attached only to the VMI into VM.spec, so they stay attached; the check in the PR description is optional. This also applies to ephemeral volumes added to running user VMs at the VMI level, such as with virtctl addvolume without --persist: they become part of VM.spec and attach again after a VM restart. virt-api now refuses VMI-level addvolume and removevolume on a VMI owned by a VM, so such volumes are added and removed through the VM. A tenant cluster whose kubernetes release is suspended or failing keeps its old kubevirt-csi, and a driver from before July 2025 cannot attach volumes there until that release reconciles, and a volume it attaches during the upgrade itself can be unplugged again. A values override that puts DeclarativeHotplugVolumes into disabledFeatureGates now fails the kubevirt render. Clusters on the default variant do not run platform migrations and should run that check before upgrading. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Declarative hotplug volumes are enabled by default for KubeVirt. * Running virtual machines keep eligible hotplugged volumes and disks when their configuration is migrated. * **Bug Fixes** * The migration avoids adding duplicate volumes or disks and does not mark the migration complete if required operations fail. * The Kubernetes CSI driver no longer uses the VMI hotplug fallback. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
86bf2ba8ec |
fix(kubernetes): let kubevirt-csi hot-unplug from an orphaned VMI (#4563)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does This lets the kubevirt-csi controller detach a volume from a VM instance that no VM owns. Right now that call is always denied. Closes #4452. The driver has a fallback for a volume that is still attached to the VMI but is no longer in the VM spec. It calls the VMI-level `removevolume` subresource. The `-kcsi` Role grants only the two VM-level subresources, so the call ends in 403. Nothing else reaches this service account with that grant. Only its own Role and ClusterRole name it as a subject, and the group bindings that cover every service account grant nothing on this subresource. At v1.9.0, `kubevirt.io:default` grants only `kubevirts`, `version` and `guestfs`. `kubevirt.io:edit` and `kubevirt.io:admin` do include the VMI subresource, but they aggregate only into the built-in `edit` and `admin` roles, and no tenant binding gives those to this account. The cozystack tenant roles aggregate on their own labels and don't list it. The fallback is still needed after the flag in #4453 goes to false. The flag only skips VMIs that a VM owns. When a VM is deleted with `propagationPolicy=Orphan`, the KubeVirt VM controller drops its finalizer and leaves the VMI running (`vm.go` at v1.9.0). The tenant node stays up with its volumes attached. When a pod moves off that node, the driver has to detach through the VMI, and on a 403 the hot-plug pod keeps holding the infra volume. Until #4453 lands, the fallback also runs for VM-owned VMIs and gets the same 403. The grant is `update` on `virtualmachineinstances/removevolume` and nothing more. The account can already remove a volume from any VM in the namespace, so it gets no new reach. virt-api also rejects the VMI-level call on a VM-owned VMI once `HotplugVolumes` is off. `virtualmachineinstances/addvolume` stays out because the driver only attaches through the VM. The new helm-unittest pins the Role's `subresources.kubevirt.io` rule to these three subresources with `update`. I haven't reproduced the 403 on a cluster. This shows it on a tenant namespace, before and after: ```bash kubectl auth can-i update virtualmachineinstances.subresources.kubevirt.io --subresource=removevolume \ --namespace <tenant-ns> --as system:serviceaccount:<tenant-ns>:<release>-kcsi ``` Upstream kubevirt/csi-driver has the same gap in `deploy/infra-cluster-service-account.yaml`. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> No UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(kubernetes): the kubevirt-csi controller can now hot-unplug a volume from a VMI that no VM owns, instead of failing with 403 Forbidden. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved Kubernetes CSI volume removal for virtual machine instances, including cases where a volume is not listed in the virtual machine specification. This helps the driver complete volume removal in scenarios where the Kubernetes virtualization API accepts the instance-level request. Existing permissions for adding and removing volumes from virtual machines remain unchanged. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
c7b8009964 |
fix(kubernetes): log kubevirt-csi temp NFS mount dir removal failure (#4562)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The kubevirt-csi node plugin now logs a warning when it cannot remove the temp dir it uses for NFS publish. Before, the error was dropped, so a dir left behind after a failed unmount was never reported. The publish path temp-mounts the export root to create `/data`, then removes the temp dir in a defer. The failure is logged, the same way the unmount failure next to it already is, and not returned. A leftover temp dir does not break the volume, and failing a publish that otherwise worked only makes kubelet retry it. A new unit test publishes an NFS volume through the fake mounter, where the removal always fails, and checks that a warning naming the dir is logged. This is independent of #4451. That PR touches the same Go module but not `node.go`. A failed unmount still leaves the NFS root mounted. That is older than this change and is tracked in #4561. Closes #4466 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> No UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(kubernetes): the kubevirt-csi node plugin now logs a warning when it cannot remove its temporary NFS mount dir instead of dropping the error ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Failed cleanup of a temporary mount directory now produces a warning with the directory path, making the issue easier to identify. The volume publish operation continues to succeed when cleanup fails. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
2af8b0c229 |
ci(build): fall back to direct, retry and cache Go module downloads (#4660)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does One transient error from proxy.golang.org is enough to fail an image build or the `Unit & controller tests` job today. The error is an HTTP/2 stream reset (`stream error ... INTERNAL_ERROR`), and nothing retries or falls back after it. This PR adds three things: a direct fallback, a bounded retry and a module cache. Image builds now fall back to fetching modules from their origin when the proxy fails. They get `GOPROXY=https://proxy.golang.org|direct` as a build arg from `hack/common-envs.mk`, and every Go builder stage declares `ARG GOPROXY`. With a pipe Go goes direct on any proxy error. With Go's default comma it does that only on a 404 or 410. The value comes from `IMAGE_GOPROXY` and not from the host's `GOPROXY`, because buildkit can't reach a mirror on localhost or a `file://` proxy. Every workflow job that runs Go on the runner sets the same value as env. Explicit module downloads in Dockerfiles are retried three times with a growing pause. The loop is inline, in one form that the contract test pins. I tried a shared script from a named build context first, but buildx refuses any `--build-context` for the four Dockerfiles still pinned to the `dockerfile:1.2` frontend. The images also build from different contexts, so no single file can be copied in. Raising those pins changes how those images build and needs a real build to check, so I left it for a follow-up. Two alpine stages, token-proxy and shadowbox's `ss-server`, left the download to `go build`. The alpine golang images have no git, so the direct fallback can't fetch for them, and they now get the same download step. The runner jobs use `hack/go-mod-download.sh` with the same bound. The unit test job now starts from a module cache instead of downloading everything. It restores `GOMODCACHE` with `actions/cache`, keyed on every `go.sum` in the tree, and then prefetches all tracked modules through the retry, so a cache miss still works. A cache saved by a PR run is visible only to that PR, but one saved on the default branch can be restored by any PR ([docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#restrictions-for-accessing-a-cache)). That is why the build cache warmer on main writes the same key before its image build. This way the first run of a PR, which is the one that flakes, starts warm. The warmer restores only an exact key, so old module versions don't pile up, and its Go steps can't fail the image build. Both runner labels are Linux, so the key matches. Both also check out under `/home/ubuntu/_work`, so `go env GOMODCACHE` should give the same cache path, but I have not seen it on the warmer's runner yet. The release jobs in `tags.yaml` and `promote-rc.yaml` get the GOPROXY env only, no cache. `hack/go-module-fetch-contract.bats` covers all three. Each test is red on main, and I checked each guard with a mutation. Closes #4591 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note ci(build): Go module downloads in image builds and CI jobs now fall back to direct fetching and retry on transient proxy.golang.org errors. The unit test job reuses a module cache written from main. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Reliability** - Go module downloads now fall back to direct access when the public proxy encounters errors. - Builds and automated checks retry failed module downloads, helping them recover from temporary network issues. - **Performance** - CI caches Go modules to reduce repeated downloads and speed up builds and tests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
94abf6940b |
feat(kubernetes): let kubevirt-csi detach without the HotplugVolumes gate (#4451)
## What this PR does This lets kubevirt-csi detach volumes without the deprecated KubeVirt `HotplugVolumes` gate. The default stays as it is, so nothing changes at runtime yet. Part of #4032. The driver got a switch upstream for exactly this, and I bumped to it. kubevirt/csi-driver#382 added `--enable-vmi-hotplug-fallback`, merged as cb4770e4e475. With it set to false, the driver no longer calls the VMI-level `removevolume` on a VMI owned by a VM. virt-api rejects that call once `HotplugVolumes` is off. Our wrapper now passes the flag to upstream and no longer has its own hot-unplug path. `unpublishHotplugVolume` was a backport of kubevirt/csi-driver#184, and the previous pin already contained that change. Upstream detaches from the VMI when the VM is gone, then waits for the VMI status to drop the volume before it returns. So the wrapper's re-check never found the volume attached. Adding the ownership check there would have been dead code. Closes #4450 The chart leaves the flag at its default of true and does not render it. The driver image in the chart is stamped only at release preparation, so between releases main runs the previous release's binary, which does not know the flag. Go's `flag` package exits on an unknown flag. Rendering `=true` would crash-loop the CSI controller for no effect. The right value depends on the KubeVirt CR, not on the tenant, so the comment next to `HotplugVolumes` in `kubevirt-cr.yaml` points at the controller Deployment. The follow-up PR drops the gate and renders the flag as false. Nothing changes on the node side. Main already pins f5de10d39556, and cb4770e4e475 is that commit plus kubevirt/csi-driver#382. A Go unit test runs an unpublish through the wrapper during a teardown, with the flag on, off with a VM-owned VMI, and off with an orphaned VMI. It checks that the flag reaches upstream and that only upstream makes the VMI-level call. Upstream's own tests cover the ownership rule. A second test holds that an NFS volume only releases the node from its CiliumNetworkPolicy and never reaches the upstream hot-unplug. A helm-unittest pins that the chart renders no such argument. I haven't built the image or run e2e. Volume attach and detach on a tenant cluster in e2e is the real check. ### Screenshots No UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(kubernetes): kubevirt-csi supports --enable-vmi-hotplug-fallback, so it can detach volumes without the KubeVirt HotplugVolumes feature gate. The flag stays true for now and behaviour is unchanged. ``` |
||
|
|
b2a5f12e73 |
build(kubernetes): run the tests of the packages the patches touch
The cluster-autoscaler and kubevirt-cloud-provider image builds ran go build only, and go build never compiles a _test.go, so nothing ran the tests of the code the patches change. Run go test in the builder stage on the packages the patches touch. The list is read from the patches themselves, so a new patch is covered without an edit to the Dockerfile. Both builder stages export GOOS and GOARCH for the target, so the tests unset them: a local make image builds amd64 and arm64 by default, and a test binary for the other architecture runs on the build host only under binfmt emulation. The first run of that step on cluster-autoscaler found the carried downscale patch out of step with the upstream tests it leaves in place: SetSize no longer refuses a size above maxSize, while one case of TestNodeGroupIncreaseSizeErrors still expected that refusal. The patch now drops that case, and carries a test for the downscale it is named for instead. DeleteNodes lowers a group one node at a time through SetSize, so with the refusal in place a group standing above its max size could not shed a node at all. The new test deletes a node from such a group and expects the replica count to drop; against the unpatched SetSize it fails with the refusal the patch removes. Scale-up past maxSize stays bounded by the core scale-up orchestrator, which skips a node group already at MaxSize. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
64f43f4f77 |
fix(computeplane): do not grant tenants the cluster admin kubeconfig
The computeplane module deploys the kubernetes chart into the tenant namespace, and that chart's dashboard Role lets the tenant's "use" subjects read <release>-admin-kubeconfig. Withholding the Secret from the tenant secrets view does not help when RBAC grants it directly, so any tenant user at "use" level or above could fetch the cluster-admin credential of a cluster the module promises to keep operator-owned. The kubernetes chart now drops that rule when the internal value withholdAdminKubeconfig is set, and the module sets it in the cluster HelmRelease's inline values. The tenant cannot reach those values: the module reads only nodeGroups from them and splices an allowlisted subset into the pool releases, and Flux applies spec.values over valuesFrom. A regular kubernetes app keeps the rule, so its dashboard can still offer the kubeconfig. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
fccb7fee06 |
test(tests): fail absence-only cases when the template renders nothing
Each of these templates is wrapped in a top-level condition, and the case asserts only that a field is absent. If a values change turns the condition off, the render is empty and the absence holds trivially, so the case passes without looking at anything. A document selector makes an empty render fail with "document not found". Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
548d33f01f |
build(images): fall back to direct and retry Go module downloads
A transient proxy.golang.org HTTP/2 stream reset fails an image build outright: Go's default GOPROXY falls back to direct only on a 404 or 410. Image builds now get GOPROXY=https://proxy.golang.org|direct as a build arg from hack/common-envs.mk, declared by every Go builder stage, so any proxy error falls back to fetching from the origin. The value comes from IMAGE_GOPROXY rather than the host's GOPROXY, which may name a mirror that buildkit cannot reach. The alpine golang images carry no git, so in their stages a module hosted on GitHub cannot be fetched directly and the retry is the cover. Explicit `go mod download` steps retry three times with a growing pause. The loop is inline, in one form the contract test pins: the images build from different contexts, so no shared script can be copied in, and a named build context in the shared buildx arguments makes buildx refuse every Dockerfile pinned to a frontend older than dockerfile:1.4. The two alpine stages that left the download to `go build` get the same download step, because without git neither the fallback nor a retry covered them. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
d6b38f0a79 |
fix(kubernetes): fail the OIDC bootstrap Job when the tenant CA is missing (#4232)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does In `spec.oidc.mode: System`, the post-install/post-upgrade bootstrap Job patches the chart-owned `<release>-oidc-kubeconfig` Secret with a real kubeconfig built from the tenant admin kubeconfig's CA data. When that CA data was empty, the script logged a WARNING and continued, then exited 0. The hook was satisfied, the HelmRelease reported `Ready`, and the Secret was left holding the install-time placeholder kubeconfig, which cannot authenticate. This PR makes the Job fail (`exit 1`) instead, so the failure surfaces as a red install and can be retried (the Job already has `backoffLimit: 6` and `restartPolicy: OnFailure`) rather than reported as a falsely healthy release. This is the smallest of the three directions raised on the linked issue, and matches how the neighbouring failure mode in the same script already behaves (a failed `kubectl apply` of the ClusterRoleBindings already fails the Job under `set -euo pipefail`). Behaviour for `mode: None` and `mode: CustomConfig` is unchanged, and the common case — CA data present — runs the identical patch commands as before. A regression test was added to `packages/apps/kubernetes/tests/oidc_test.yaml` asserting the rendered Job script contains the `exit 1` on empty `CA_DATA` and no longer contains the old warn-and-continue text. Confirmed it fails against the pre-fix template and passes against the fix. Validation: - `helm unittest .` in `packages/apps/kubernetes` — 150/150 tests pass (up from 149 before the new test). - `helm template ... --show-only templates/oidc-rbac-job.yaml` with representative `System`-mode values renders valid YAML; the extracted bootstrap script passes `bash -n` and `shellcheck -s sh` introduces no new warnings versus the pre-change script. - `make generate` was not run/needed — this change touches neither `values.yaml`, `values.schema.json`, `Chart.yaml`, nor `README.md`. - Not run: a live cluster reproduction (Kamaji-issued admin kubeconfig actually missing CA data at Job run time). The defect is provable from the script logic itself and the added unit test demonstrates the fail-then-pass behaviour; there is no live cluster available in this environment. ### Screenshots Not applicable — no UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: This change is confined to the shell logic inside `packages/apps/kubernetes/templates/oidc-rbac-job.yaml`'s post-install hook Job (a `Values.oidc.mode: System`-only failure path) and its accompanying helm-unittest test. It does not add, rename or remove a package, a `values.yaml` key, a CRD, a namespace, a release-owned object name, or any other surface the trigger map lists, so no downstream repository is affected. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(kubernetes): a System-mode OIDC install no longer reports Ready while the tenant OIDC kubeconfig Secret still holds the install-time placeholder — the bootstrap Job now fails (and retries) instead of only logging a warning when the tenant cluster's CA data isn't available yet. ``` Fixes #3765 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * OIDC setup now fails with an error if the tenant admin kubeconfig is missing certificate authority data, rather than leaving the OIDC kubeconfig Secret at its install-time placeholder. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
4fe45eba32 |
fix(kubernetes): fail the OIDC bootstrap Job when the tenant CA is missing
In mode=System the post-install hook Job patches the chart-owned <release>-oidc-kubeconfig Secret with a real kubeconfig built from the tenant admin kubeconfig's CA data. When that CA data was empty the script only warned and exited 0, so the hook was satisfied, the HelmRelease reported Ready, and the Secret kept the install-time placeholder kubeconfig, which cannot authenticate. The failure only showed up when a user tried to use the downloaded kubeconfig. Exit 1 instead, so the install fails visibly and is retried. The Job already has backoffLimit: 6 and restartPolicy: OnFailure, and a failed ClusterRoleBinding apply in the same script already fails the Job the same way. Assisted-by: LLM Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com> |
||
|
|
5be222ad75 |
docs(kubernetes): say the token's user needs the pool privileges too
A privilege-separated Proxmox token never exceeds its user's permissions. Granting the README's roles to the token alone left the CSI and CCM tokens with an empty effective set on the pool and the storage; granting the same roles to the user made the token's own ACL the effective one. Seen on Proxmox VE 9.2.20 while running the storage pair against a pool-scoped user, and worth one sentence next to the table so nobody repeats it. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
|
36b04218d7 |
test(kubernetes): assert where the hypervisor token is and where it is not
Every assert here was written against the mutation it exists for, and each was checked to turn red on it and green without: putting the controller back in the tenant, giving the tenant a non-empty config.clusters, pointing a sidecar at the wrong kubeconfig, restoring the blanket toleration, hardcoding the driver's default owner id, dropping capmox mode, and letting the controller ignore proxmox.csi.enabled. The negations carry `any: true`. Without it a negated containsDocument passes whether or not the document is in the render. Two of the cases are about objects rather than values: the tenant release must carry no valuesFrom at all, and two clusters must derive different volume owner ids. Both are properties a reader can check against the rendered output, which is why they are asserted that way rather than by reading the templates. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
5ab36023e0 |
fix(kubernetes): release Proxmox volumes before the driver that frees them goes
Deleting a tenant cluster left its PVC disks on the hypervisor. The pre-delete
hook deletes every HelmRelease labelled with the cluster in Step 2, the CSI
driver among them, and nothing before that deleted the tenant's PVCs — a volume
is freed only by the driver answering DeleteVolume. capmox then destroys the
worker VMs, and Proxmox frees only what the destroyed VMID owns, which these
volumes are not. They stayed on storage with the tenant's data in them, and the
PV objects that could have named their owner went with the control plane.
Step 1b deletes the claims while the driver is still up, and it deletes the
claims before the Pods that hold them: a claim being deleted is one the
scheduler places no new Pod on, so a Pod that a StatefulSet or Deployment brings
back cannot mount it again and hold it to the backstop. The Pods go second: a
claim in use carries the kubernetes.io/pvc-protection finalizer, so it clears
only once nothing mounts it, and the Pods are going with the cluster in any
case. Then it waits for the PVs rather than the PVCs, because the claim
disappears as soon as its finalizer clears while the disk is freed only when the
PV is gone.
The Pods holding a claim are found by listing every Pod with the claims it
mounts and matching in the shell. A JSONPath filter on the claim name cannot do
it: with a Pod that mounts two claims its left side yields two values, client-go
refuses the whole expression ("can only compare one element at a time"), and no
Pod in that namespace would be found for any claim.
A kubeconfig that exists is not an apiserver that answers, and listing claims
against one that does not comes back empty — the clean case. Step 1b asks
/readyz first and sends an apiserver that stays silent through the backstop, so
the summary carries it. The probe is bounded by `timeout` rather than by
--request-timeout: a non-zero --request-timeout on an in-cluster kubectl call
diverts it to localhost:8080, which is why hack/check-hook-request-timeout.bats
refuses the flag in hook templates, and the check cannot tell this call, which
carries its own kubeconfig, from one that does not.
Best effort, and loud about it: a tenant that never came up has no apiserver to
ask, and a teardown must not wedge on that, but a timeout is reported through
the same backstop the other steps use.
The hook's Role gains one Secret by name — the tenant's admin kubeconfig, which
this step needs. Without it the step takes its no-kubeconfig branch and reports
that there is nothing to release while the volumes stay behind.
Step 5 no longer fails the whole hook where CDI is not installed. It deletes
DataVolumes, which are a KubeVirt-path component, and kubectl exits non-zero on
an unknown resource type; under set -e that failed the teardown after every
earlier step had already run, leaving the cluster half removed and the hook
reporting failure.
Verified on a throwaway tenant: a PVC bound and mounted by a running Pod, then
`helm uninstall`, and the volume freed by the driver rather than by hand.
delete_hook_test.yaml pins Step 1b and the Role's admin-kubeconfig entry on
proxmox and their absence on kubevirt, the /readyz probe, and the order of the
claim and Pod deletes.
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
|
||
|
|
d902421f47 |
feat(kubernetes): deploy the Proxmox cloud-controller-manager with the cluster
On this substrate the worker kubelet runs with --cloud-provider=external, which taints every node node.cloudprovider.kubernetes.io/uninitialized until a cloud provider clears it. Without a cloud-controller-manager the taint is permanent: nothing but tolerating DaemonSets schedules, and the CSI node plugin — which reads topology.kubernetes.io/region and zone off its own Node and exits fatally without them — never starts. It runs in the management cluster for the same reason as the CSI controller beside it: it authenticates to the hypervisor, and the tenant is cluster-admin of its own cluster. Its whole job is to write on the tenant's Nodes, and it does that over the tenant's admin kubeconfig from out of cluster, which also means there is no tenant-side RBAC left to install. capmox mode, not the default one. The default resolves a node to its VM by name and writes providerID as proxmox://<node>/<vmid>, while capmox reads the CAPI Machine and writes proxmox://<SMBIOS UUID> — the form capmox itself puts on the Machine. With the default the Node and its Machine carry two different identifiers for the same VM. The tolerations are spelled out rather than a blanket `operator: Exists`. That one also matches node.kubernetes.io/unreachable:NoExecute with no tolerationSeconds, which stops admission adding the bounded 300s one and lets the Pod outlive the Node it runs on. `proxmox.insecure` defaults to false: the controllers verify the hypervisor's certificate unless an operator turns that off, and both run here, so the value describes what the management cluster trusts, not the tenant. The README gains a section on what the two tokens can reach — the controller acts on whatever VMID and volume handle the tenant's objects name — and on the per-tenant Proxmox scope that contains it: a storage of the tenant's own and one ACL on the resource pool proxmox.pool puts its workers in, which follows every VM capmox creates — VM.Audit there for the CCM too, which deletes a NotReady Node whose VM it cannot see. ccm, csi and insecure are optional in the schema. Each has a default, and what the substrate needs is enforced at render time — a proxmox cluster without ccm.credentialsSecretName does not render — so a Kubernetes object written before these fields existed stays valid, and adding them is not a breaking change to the API. The kubelet flag is asserted on both substrates: present on proxmox, absent on kubevirt, where nothing would ever clear the taint it causes. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
93c89c8bf6 |
feat(kubernetes): add the Proxmox CSI driver, with its controller in this cluster
Tenant clusters on the proxmox substrate get persistent volumes backed by the
hypervisor's storage. The driver is sergelogvinov/proxmox-csi-plugin, vendored
as a chart that reproduces byte-for-byte from `helm pull`.
It is deployed in halves, and which half goes where is the whole design. The
controller authenticates to Proxmox, so it runs HERE, in the tenant's namespace
of the management cluster, and drives the tenant over its admin kubeconfig —
the same shape templates/csi/deploy.yaml already uses for kubevirt-csi. The
tenant gets the node plugin, the CSIDriver object and the storage classes; its
DaemonSet mounts no credentials.
Putting the controller in the tenant would hand the token to the tenant. The
resource definition lists the admin kubeconfig in secrets.include, so the
tenant is cluster-admin of its own cluster and can read any Secret there, and
the chart writes the driver's config.yaml — token included — wherever it is
installed. The privileges the driver documents are granted on `/`, so that
token reaches every VM and every volume on the Proxmox cluster, not just this
tenant's.
Three consequences of the split are handled here rather than discovered later:
- The vendored chart has no switch for its controller, so the tenant release
sets replicaCount: 0. The Deployment object still renders; no Pod does.
- Its Secret template is gated on a non-empty config.clusters, so an empty
list is what keeps the credential out.
- CSIStorageCapacity is published by the provisioner sidecar, which now runs
here and owns those objects through a Pod that does not exist in the
tenant. Left on, CSIDriver.storageCapacity=true and the scheduler refuses
every Pod with "node(s) did not have enough free storage". The chart
hardcodes that field, so a kustomize postRenderer patches it rather than
forking the chart.
The controller follows proxmox.csi.enabled, like the tenant release: with the
driver switched off there is nothing for it to serve and no token to hold.
Volumes are owned per tenant. The driver names them vm-<controllerVmID>-pvc-…
and defaults the id to 9999; on shared storage that made every tenant's volumes
indistinguishable to an operator and to a VM-level ACL. Each cluster derives its
own from its release name.
What this does not do is make one tenant unable to reach another's volumes: a
tenant can still create a static PV naming a foreign volumeHandle. That needs a
per-tenant Proxmox scope — a dedicated storage and a VM ACL per tenant — which
is a deployment requirement rather than a chart change.
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
|
||
|
|
53829ea109 |
fix(kubernetes): reconcile KCCM EndpointSlices when tenant node VMIs change (#3500)
## What this PR does Adds `patches/vmi-informer.diff` to the `kubevirt-cloud-provider` image of the `kubernetes` package. The file is named after what it does: no upstream pull request carries this change yet (kubevirt/cloud-provider-kubevirt#382 covered the same desync and was closed as rotten), so a number would point at an unrelated upstream PR and invite someone bumping the pin to drop it. The `kubevirteps` controller derives the infra EndpointSlices of a LoadBalancer Service from the VMIs backing the tenant nodes (`getDesiredEndpoints`, and the all-VMIs fallback added by `379.diff`). But it registers informers only on infra Services, tenant EndpointSlices and infra EndpointSlices, and both factories are built with a resync period of `0`. VMI addresses are resolved with a one-off `Get` during a reconcile. Consequently a VMI appearing, disappearing or changing its address triggers no reconciliation, and no periodic relist ever catches up. The EndpointSlices keep pointing at VMIs that no longer exist and the LoadBalancer blackholes the share of the inbound connections hashed onto those dead addresses — sporadic inbound timeouts, with nothing wrong on the tenant side. Services whose tenant EndpointSlice churns along with its pods recover on their own, because that churn produces an event. Cilium Gateway API Services do not: their tenant slice is a static sentinel (`192.192.192.192`, no `NodeName`), so nothing ever forces a recompute. Seen in production on a cluster whose GPU MachineDeployment is rolled by the cluster-autoscaler: 5 of the 9 endpoints of each of the 3 gateways pointed at destroyed VMs (13 failures out of 20 requests). Worth noting for anyone debugging this: the KCCM looks dead (no log for hours) while its lease is still being renewed every second. The process is healthy, it is simply never woken up. The patch: - adds an informer on the infra `VirtualMachineInstances` that requeues the LoadBalancer Services of this cluster (those labelled `cluster.x-k8s.io/cluster-name=<clusterName>`, the ones `reconcile` does not skip) on add/delete, and on updates filtered by a fingerprint of what actually shapes an endpoint: the VMI's cluster label, `Status.NodeName`, `Status.Phase`, and the address of the `default` interface; - gives both informer factories a 5 minute resync, as a safety net if an event is ever missed; - scopes the all-VMIs fallback to the VMIs labelled `cluster.x-k8s.io/cluster-name=<clusterName>`. An infra namespace can hold several tenant clusters as well as standalone `VirtualMachines`, and their VMIs do not serve this Service's NodePort, so selecting them adds endpoints that blackhole traffic. The fallback widens to every VMI only when none of them carries the label, so an environment that does not label them does not lose all of its endpoints; once any VMI is labelled, an empty selection means this cluster has no node right now and is not widened to another cluster's VMIs. No RBAC change: the KCCM role already grants `watch` on `virtualmachineinstances`. On `main`, the Gateway scenario reaches the all-VMIs fallback without any other KCCM patch: Cilium 1.19.5 (shipped since v1.6.0) sets `kubernetes.io/service-name` on the tenant slice of a `cilium-gateway-*` Service (cilium/cilium#46242), so the existing lookup finds the sentinel slice and falls back on its missing `NodeName`. This is why #3374, which added that lookup to the KCCM, was closed. On v1.5.x and earlier, which ship Cilium 1.19.3, a Gateway Service finds no tenant slice at all, so it has no endpoints to go stale in the first place. ## Testing - `341 + 379 + vmi-informer` applies onto the pinned `a0acf33`; the Dockerfile steps (`go get`, `go mod tidy`, `go mod vendor`, `-mod=vendor` build of the manager) pass, `go vet` and `gofmt` are clean. - The patch adds 6 specs, and the suite runs 21 of 21 green: fallback scoped to the cluster, fallback preserved when no VMI is labelled, no widening when only another cluster's VMIs are present, unlabelled VMIs ignored once some are labelled, VMI add/delete requeues the Service of this cluster and not the one of another cluster, and VMI updates driven through the fake dynamic client: an annotation-only update does not requeue, while a change of phase, node, address, or cluster label does. - Mutation-tested, each one turns the suite red: always requeuing on update; dropping the phase, the node, the address or the cluster label from the fingerprint; requeuing every Service of the namespace; dropping the delete requeue. - Under `-race`, the suite reports an intermittent data race at `kubevirteps_controller_test.go:820` (a `PrependReactor` racing an informer list). It reproduces identically on `341 + 379` without this patch, and the new specs pass on every `-race` run. - Validated on a live cluster with an image built from an earlier revision of this patch (same informer and fallback, before the cluster-scoped requeue and the label in the fingerprint): endpoint added in under 2s when a VMI appears, dead address removed in 33s when it is deleted. Before the patch the dead address stayed forever. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff is one patch file of the KCCM image: it touches no values, schema, CRD, `hack/` tooling, namespace, telemetry metric or node prerequisite, so the trigger map comes out empty. ### Release note ```release-note fix(kubernetes): the KCCM now reconciles the LoadBalancer EndpointSlices when the VMIs backing the tenant nodes change, instead of keeping stale addresses that blackhole part of the inbound traffic ``` |
||
|
|
924d28cf3d |
feat(kubernetes): default gpu-operator to an OS-provided driver on Talos (#3585)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Closes the gpu-operator half of #3563 — gap 3 (the operator cannot validate a driver it did not install) and gap 4 (nothing routes CUDA libraries to a tenant pod). Tenant clusters have run Talos workers since 1.6.0, where the NVIDIA driver comes from a system extension baked into the worker image rather than from the operator's driver container. The addon default still described the Ubuntu shape, so a GPU node group came up with the operator fighting the OS for the driver and no path at all for the CUDA libraries. Three changes, each closing one silent failure: | Change | Without it | |---|---| | `driver.enabled: false`, `toolkit.enabled: false` | the driver container loads its own module and clashes with the extension's; the toolkit container rewrites `/etc/containerd/config.toml`, which Talos owns and which already registers the `nvidia` runtime with `enable_cdi = true` and `cdi_spec_dirs = ['/run/cdi']` | | `validator.driver.env`: `DRIVER_ROOT=/`, `NVIDIA_DRIVER_ROOT=/`, `DISABLE_DEV_CHAR_SYMLINK_CREATION=true` | the driver validator looks for a driver-container install, finds none, never writes its sentinel, and the device plugin stays blocked behind `toolkit-validation` — the loop on `Attempting to validate a driver container installation` (**gap 3**) | | `devicePlugin.env`: `DEVICE_LIST_STRATEGY=cdi-annotations`, `NVIDIA_CTK_PATH=/usr/local/bin/nvidia-ctk`, plus `DEVICE_ID_STRATEGY`, `NVIDIA_DRIVER_ROOT`, `CDI_ANNOTATION_PREFIX` | tenant pods carry no `runtimeClassName` and did not need one on Ubuntu, where the toolkit made `nvidia` the default containerd runtime. Talos does not, so CDI is the only route: the pod gets the device and no CUDA libraries (**gap 4**) | Every value is the configuration validated on a production Talos GPU node group, minus the parts that only mattered while the driver container was in play. No DaemonSet, no new image, no new package — the fix is values. ### The removed default The previous default fed `NVreg_NvLinkDisable=1` to the driver container through the `nvidia-kernel-module-params` ConfigMap. With that container off the ConfigMap is mounted by nothing, so this removes the default. On Talos the parameter belongs to the worker machine config, which the `kernelModules` field of the worker pool writes: that field is top-level in the KubernetesNodes app (`packages/apps/kubernetes-nodes`), added by #3571, and its automatic set for a pool holding an `nvidia.com/*` GPU already loads `nvidia` with `NVreg_NvLinkDisable=1`. The wrapper chart keeps the ConfigMap feature for an operator who re-enables the driver container through `valuesOverride`. A test pins the absence so the dead default cannot come back and offer a second, inert source for the same setting. ### Relationship to #3571 Independent files, no conflict, but only useful together: this makes `driver.enabled: false` correct, and #3571 is what actually loads the modules the extension ships. Merged alone, this changes a GPU node group from "operator fights the OS" to "operator stays out of the way and the modules still are not loaded" — better, but not yet working. #3571 alone leaves the operator fighting for the driver. Either order is safe; both are needed for a GPU node group to work. ### What is verified, and what is not Verified here: the rendered HelmRelease values, that `valuesOverride` still wins over each new key, and that the removed default is gone. The individual values are the ones running in production today, applied by hand through `addons.gpuOperator.valuesOverride`. Not verified here: this exact default exercised end-to-end as the chart's own output. That needs a Talos GPU node group whose modules load, which needs #3571 released. I would rather say so than imply a green e2e that does not exist — happy to hold this as draft until I have run a single-node L40S cluster against the chart default, and I will report the result on this PR either way. ### Not covered HAMi. When `addons.hami.enabled` the operator's device plugin stays off and HAMi serves the GPUs; HAMi's own plugin needs equivalent CDI wiring on Talos, which this default does not attempt. Flagging rather than guessing — the production cluster this was derived from runs HAMi disabled, so I have no validated configuration for that path. Worth its own issue if a maintainer confirms the shape. Also not in scope, and unchanged from the merge base: `talos.installerRepository` and `talos.version` reach the worker machine config unvalidated (being tracked separately by @lexfrei), and the `nvidia-operator-validator` chain for the *management* cluster's own passthrough/vgpu variants, which never load the host driver at all. ### Screenshots No UI changes. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/643 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(kubernetes): The GPU Operator addon of tenant Kubernetes clusters now defaults to a driver provided by the operating system, which is the shape of the Talos workers these clusters have run since 1.6.0. `driver` and `toolkit` are disabled (the driver comes from a Talos system extension, and containerd configuration belongs to the machine config), the driver validator is pointed at the host root so it stops looping on "Attempting to validate a driver container installation", and the device plugin is switched to CDI (`DEVICE_LIST_STRATEGY=cdi-annotations`) so tenant pods receive the CUDA libraries — previously they received the GPU device and nothing else, since Talos does not make `nvidia` the default containerd runtime. The previous default, which fed `NVreg_NvLinkDisable=1` to the operator's driver container through a ConfigMap, is removed because that container is disabled; on Talos the parameter is carried by the worker machine config, through the `kernelModules` field of the worker pool in the KubernetesNodes app (`packages/apps/kubernetes-nodes`), which loads `nvidia` with `NVreg_NvLinkDisable=1` by default on a pool holding an `nvidia.com/*` GPU. Operators who overrode `addons.gpuOperator.valuesOverride` to work around this by hand can drop those overrides. NOTE: an override that sets `devicePlugin.env` replaces the list rather than merging it, so restate the full list when overriding it or CDI stops working. HAMi is not covered: when `addons.hami.enabled` the operator's device plugin stays off and HAMi's own plugin needs equivalent CDI wiring, which this change does not attempt. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Configuration** * GPU workloads on Talos workers now use host-provided NVIDIA drivers and tooling by default; the GPU Operator’s driver and toolkit are disabled. * Device discovery uses CDI annotations and UUID-based device IDs. When HAMi is enabled, the Operator’s device plugin is disabled. * Host-root validation no longer creates `/dev/char` symlinks. * Driver kernel-module configuration is no longer created by default. * **Documentation** * Clarified when kernel-module configuration applies and what settings are needed to re-enable the driver. * Noted that `valuesOverride` replaces lists wholesale. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
1f9b1b85db |
fix(kubernetes): reconcile KCCM EndpointSlices when tenant node VMIs change
The kubevirteps controller derives the infra EndpointSlices of a LoadBalancer Service from the VMIs backing the tenant nodes, but it watches no VMI and its informers had no resync period. A VMI appearing, disappearing or changing its address therefore triggered no reconciliation, and no periodic relist ever caught up: the EndpointSlices kept pointing at VMIs that no longer existed and the LoadBalancer blackholed the share of the inbound connections hashed onto those dead addresses. Services backed by a tenant EndpointSlice that churns with its pods recovered on their own. Cilium Gateway API Services did not: their tenant slice is a static sentinel, so nothing ever forced a recompute. Observed in production on a cluster whose GPU MachineDeployment is rolled by the cluster-autoscaler: 5 of the 9 endpoints of every gateway pointed at destroyed VMs. Add an informer on the infra VirtualMachineInstances that requeues the LoadBalancer Services of this cluster, ignoring the updates that cannot change an endpoint, and give the informers a 5 minute resync as a safety net. Services of other tenant clusters sharing the namespace are not requeued, since reconcile would skip them. Also scope the all-VMIs fallback to the VMIs labelled with the name of the cluster being reconciled: an infra namespace can hold several tenant clusters and standalone VirtualMachines, whose VMIs do not serve the Service NodePort. The fallback widens to every VMI only when none of them carries the label, so that environments that do not label them keep their endpoints. Once any VMI is labelled, an empty selection means this cluster has no node right now, and widening it would publish the addresses of another cluster. The patch is named after what it does rather than after an upstream pull request, because none carries this change yet. Signed-off-by: Mattia Eleuteri <mattia@hidora.io> Assisted-by: LLM |
||
|
|
669fb2416a |
feat(kubernetes): default gpu-operator to an OS-provided driver on Talos
Closes the gpu-operator half of #3563. Tenant clusters have run Talos workers since 1.6.0, where the NVIDIA driver comes from a system extension baked into the worker image rather than from the operator's driver container. The addon default still described the Ubuntu shape, so a GPU node group came up with the operator fighting the OS for the driver and no path at all for CUDA libraries to reach a tenant pod. Three things change, each fixing one silent failure: driver.enabled and toolkit.enabled go false. The driver container would load its own module and clash with the extension's; the toolkit container would rewrite /etc/containerd/config.toml, which Talos owns and which already registers the nvidia runtime with enable_cdi = true and cdi_spec_dirs = ['/run/cdi']. validator.driver.env points DRIVER_ROOT and NVIDIA_DRIVER_ROOT at / and disables /dev/char symlink creation. The driver validator otherwise looks for a driver container install, finds none, never writes its sentinel, and leaves the device plugin blocked behind toolkit-validation — the loop on "Attempting to validate a driver container installation". devicePlugin.env selects cdi-annotations and points NVIDIA_CTK_PATH at the host's nvidia-ctk. Tenant pods carry no runtimeClassName and did not need one on Ubuntu, where the toolkit made nvidia the default containerd runtime; Talos does not, so CDI is the only route for the CUDA libraries. DEVICE_ID_STRATEGY, NVIDIA_DRIVER_ROOT and CDI_ANNOTATION_PREFIX are set and tested alongside it: CDI device names key on the UUID strategy, the plugin resolves the driver from NVIDIA_DRIVER_ROOT, and the annotation prefix has to be the one containerd reads, so a regression in any of them breaks device resolution on its own. The previous default fed NVreg_NvLinkDisable=1 to the driver container through the nvidia-kernel-module-params ConfigMap. With that container off the ConfigMap is mounted by nothing, so the default is removed. On Talos a module parameter belongs to the worker machine config, which the kernelModules field of the KubernetesNodes app (packages/apps/kubernetes-nodes, #3571) writes. The wrapper chart keeps the ConfigMap for an operator who re-enables the driver container. Every value here is the configuration validated on a production Talos GPU node group, minus the parts that only mattered while the driver container was in play. Not covered: HAMi. When addons.hami.enabled the operator's device plugin stays off and HAMi serves the GPUs, and HAMi's own plugin needs equivalent CDI wiring on Talos that this default does not attempt. Refs: #3563 Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
2b7e0db40f |
fix(kubernetes): let kubevirt-csi hot-unplug from an orphaned VMI
The driver falls back to the VMI-level removevolume subresource when a volume is still attached to the VMI but no longer in the VM spec. The -kcsi Role grants only the VM-level subresources, and nothing else bound to the service account covers the VMI one: kubevirt.io: default grants no subresource of it, and the kubevirt.io:edit and admin roles that do are aggregated only into the built-in edit and admin roles, which the tenant namespace never binds to this account. Every such call ends in 403, the unpublish retries until it fails, and the hot-plug pod keeps the infra volume attached. The call stays needed once the driver stops falling back for a VM-owned VMI (kubevirt/csi-driver#382), because it still makes it for a VMI no VM owns. Deleting a VM with orphan propagation leaves its VMI running, since the KubeVirt VM controller then drops its finalizer without stopping the VMI, so the tenant node stays up and its volumes stay attached until the driver detaches them. The grant adds no reach the account lacks: it can already remove a volume from any VM in the namespace, and virt-api refuses the VMI call on a VM-owned VMI once the HotplugVolumes gate is off. virtualmachineinstances/addvolume stays ungranted, as the driver only attaches through the VM. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
3e91da913e |
fix(kubernetes): log kubevirt-csi temp NFS mount dir removal failure
The NFS publish path temp-mounts the export root to create /data and migrate old files, then removes the temp dir in a defer that dropped the error. When the unmount before it fails, the removal fails on a busy mount point and the leftover dir under the node's temp dir went unreported. The failure is logged rather than returned: a leftover temp dir does not make the volume unusable, and failing an otherwise good publish would only make kubelet retry it. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |