mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 05:25:14 +09:00
Aleksei Sviridkin
76b3652297
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does A node drain now completes when a VM on that node cannot live-migrate. The VM is shut down and started again instead of holding the drain forever. VMs that can migrate still migrate. The change is the cluster-wide `evictionStrategy` in the KubeVirt CR, from `LiveMigrate` to `LiveMigrateIfPossible`. The drain is held by virt-api's eviction webhook, not by a PodDisruptionBudget. With `LiveMigrate` and a VMI that is not live-migratable, the webhook denies every eviction with HTTP 429 ([pod-eviction-admitter.go#L147-L150](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-api/webhooks/validating-webhook/admitters/pod-eviction-admitter.go#L147-L150)). `kubectl drain` retries a 429 until `--timeout`, which is infinite by default. KubeVirt v1.9.0 does not create PDBs for VMIs at all; its disruption-budget controller only deletes old ones ([disruptionbudget.go#L353-L406](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/drain/disruptionbudget/disruptionbudget.go#L353-L406)). With `LiveMigrateIfPossible` the same webhook lets the eviction through for such a VMI ([L152-L155](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-api/webhooks/validating-webhook/admitters/pod-eviction-admitter.go#L152-L155)). The VM then comes back according to its `runStrategy`. RWO disks are not the only case. A VM with a passthrough GPU is not migratable either, unless the `VGPULiveMigration` gate is on and the GPU is a mediated device ([vm.go#L1310-L1319](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-handler/vm.go#L1310-L1319)). The gate is off in Cozystack, so any `vm-instance` with `gpus` set blocked the drain of its node until now. In KubeVirt a drain and a launcher update go through the same eviction. After a KubeVirt upgrade the workload updater live-migrates outdated VMIs that can migrate and evicts the rest, because `workloadUpdateMethods` lists `Evict` as the fallback ([workload-updater.go#L421-L424](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/workload-updater/workload-updater.go#L421-L424)). The eviction call ([L614-L620](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/workload-updater/workload-updater.go#L614-L620)) hits the same webhook as a drain. So a `vm-instance` VM that cannot migrate either blocks both, which is what happens today, or is restarted by both, which is what happens after this PR once the VM runs under the new strategy. This has a cost. A non-migratable VM now restarts on a drain. That covers VMs on RWO storage and VMs with GPU or host device passthrough. The webhook does not look at who asked for the eviction, so this holds for any eviction through the API, not only a drain. A VM whose disk is bound to the drained node, for example on a LINSTOR class without DRBD, stays down until the node is back. The upgrade itself restarts nothing, and existing VMs switch only after their next restart. KubeVirt writes the cluster strategy into the VMI spec when the VMI is created ([defaults.go#L208-L211](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/defaults/defaults.go#L208-L211)), and the webhook reads that field first ([migrations.go#L116-L122](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/util/migrations/migrations.go#L116-L122)). A VMI that was running before the upgrade keeps `LiveMigrate`, so a drain still waits on it and the updater's eviction still gets 429. After one restart the VMI carries `LiveMigrateIfPossible`. From then on every KubeVirt bump shuts it down through the updater, in batches of 10 every minute, the `batchEvictionSize` and `batchEvictionInterval` set in the CR ([kubevirt-cr.yaml#L139-L140](https://github.com/cozystack/cozystack/blob/25d4a8c951b9fa7249b56ae79328d2cfd4f3a9c0/packages/system/kubevirt/templates/kubevirt-cr.yaml#L139-L140)). Reading the source, a CPU socket increase on such a VM under `vmRolloutStrategy: LiveUpdate` goes the same way, since the `VCPUChange` condition sends it to the updater ([L320-L325](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/workload-updater/workload-updater.go#L320-L325)). I have not run that one. I kept `Evict` even so, because `kubernetes-nodes` workers depend on it. They set `evictionStrategy: External`, and for that strategy the webhook marks the VMI for evacuation instead of evicting it ([pod-eviction-admitter.go#L156-L157](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-api/webhooks/validating-webhook/admitters/pod-eviction-admitter.go#L156-L157)). CAPK sees the mark on a VMI that cannot migrate, drains the tenant node and deletes the VMI ([machine.go#L414-L454](https://github.com/kubernetes-sigs/cluster-api-provider-kubevirt/blob/v0.1.10/pkg/kubevirt/machine.go#L414-L454)). The worker VM has `runStrategy: Always`, so KubeVirt starts it again on the new launcher. That is how a GPU worker or any other non-migratable worker gets the new launcher. Without `Evict` those workers would stay on the old launcher for good. Worker behaviour does not change with this PR, and the new test pins the method list so the fallback is not dropped later. `vm-instance` renders no `evictionStrategy` and has no value for it, so a tenant has no way to opt out of either restart. KubeVirt would honour a per-VM value over the cluster one. I filed #4419 to expose it. I picked this over the options in the issue because KubeVirt already decides migratability per VMI at eviction time. The platform CR does not need to know anything about storage, and a chart shipping a VM does not have to remember an override. Deriving the strategy from the disk access mode in a chart would also miss the GPU case. To get the new drain behaviour for a non-migratable VM that is already running, restart it once after the upgrade. Until then a drain of its node still blocks, as it does today. The comment in the CR is on the same line as the key on purpose. `make update` inserts blocks right before the `evictionStrategy:` line, so a comment above the key would get separated from it. A VM with `runStrategy: RerunOnFailure`, `Manual` or `Once` may stay off after an eviction. If the guest shuts down cleanly on the ACPI request within its grace period, the VMI ends `Succeeded`. `RerunOnFailure` restarts only a failed VMI, and `Manual` and `Once` restart nothing on their own ([vm.go#L1039-L1149](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go#L1039-L1149)). `Once` stays off even after an unclean stop. Only the default `Always` brings the VM back in every case. What I did not verify. I have not drained a node with such a VM on a live cluster, and I have not watched the updater restart them during an upgrade. Everything above comes from the KubeVirt v1.9.0 and CAPK v0.1.10 source. I did not check how `talosctl upgrade` handles a drain that keeps getting 429. I also did not check how the cluster autoscaler treats a node with a non-migratable VM now that its launcher pod can be evicted. The pod has no `safe-to-evict` annotation, so for a VM on node-local storage that likely comes down to the autoscaler's `skip-nodes-with-local-storage` setting. Finally, I did not check whether the upstream `VMCannotBeEvicted` alert is deployed in Cozystack. For non-migratable VMs it stops firing after this change, since they are evictable now. Closes #4360 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> No UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(kubevirt)!: VMs that cannot live-migrate no longer block a node drain. The cluster-wide eviction strategy is now LiveMigrateIfPossible instead of LiveMigrate. VMs that can migrate still migrate. VMs that cannot, for example with a ReadWriteOnce disk or GPU or host device passthrough, are shut down and started again by their runStrategy. This applies to any pod eviction, not only a drain. A VM whose disk is bound to the drained node stays down until the node returns. A VM that is already running keeps the old strategy until its next restart, so restart non-migratable VMs once after upgrading if their nodes need to drain. The KubeVirt workload update uses the same eviction: once a non-migratable vm-instance VM runs under the new strategy, every later KubeVirt upgrade shuts it down, 10 VMs per minute. A VM with runStrategy RerunOnFailure, Manual or Once that shuts down cleanly on eviction stays off until it is started again. kubernetes-nodes workers are not affected. ```
Cozystack
Cozystack is a free and open-source platform and framework for building clouds.
Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.
Project documents
| Document | What it covers |
|---|---|
| Governance | How decisions are made and who makes them |
| Contributor Ladder | How to grow from contributor to maintainer |
| Maintainers | Who maintains the project |
| Code of Conduct | What we expect of each other |
| Security Policy | Reporting a vulnerability, and which releases get fixes |
| Adopters | Who runs Cozystack in production |
| Roadmap | Where the project is going |
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.