mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 14:02:56 +09:00
Aleksei Sviridkin
6735277377
## What this PR does Adds `nodeGroups.<name>.podCpuRequest` to the kubernetes chart, rendering into `domain.resources.requests.cpu` on the worker VM, and the same field to `kubernetes-nodes`. It has no default, so a cluster that does not set it renders exactly what it rendered before. It exists because `podCpuLimit` does not do what #3859 said it does. That PR shipped the limit on the claim that it moves the ceiling alone and leaves the CPU request at the value KubeVirt derives from the guest vCPU count and the cluster allocation ratio. KubeVirt does the opposite: `setDefaultResourceRequests` in `pkg/defaults` copies a declared CPU limit into the CPU request whenever the VM declares no request of its own, and it runs in the VMI mutating webhook, before the template service gets there. The request the template service computes from the vCPU count is then overwritten by the copy. So a node group that set only the limit asked the scheduler for its whole ceiling, which is what took every e2e worker out in #3860: Pending on `Insufficient cpu` before a guest booted. The request is the half that has to hold still, and the chart had no way to declare it. Worth correcting one detail of #3860 while I am here: it attributes the defaulting to Kubernetes copying limits into requests. Kubernetes does that too, but here the copy happens a layer earlier, on the VMI, which is why KubeVirt's own vCPU-derived request never appeared on the container. The first commit fixes that documentation, in both charts and in the generated schema, README and CRD descriptions that carry it. It also promised a floor that does not exist, a limit below the derived request was said to leave the container requesting more than its limit, when the copy makes the two equal whatever the limit is. The field itself is the scheduler reservation, not guest sizing, and it is usable on its own. An explicit request does not stop KubeVirt deriving the ceiling: `doesVMIRequireAutoResourceLimits` returns early only on a declared *limit*, so a lone request changes the weighting a worker gets on a contended host and leaves the ceiling where KubeVirt puts it. That asymmetry is why the docs tell you to keep a lone request at or below the guest vCPU count. Two render-time guards, both in the style of the one #3859 added. The instancetype guard is the same conflict one field over, and it is the same upstream check: `validateCPU` in `pkg/instancetype/apply` reports `domain.resources.requests.cpu` and `domain.resources.limits.cpu` in consecutive branches, so a VM carrying an instancetype and either key is refused by admission, leaving a node group with no Machine and a Kubernetes CR that explains nothing. The second guard refuses a request above its own limit, because that rejection otherwise lands on API server validation of the worker's Pod, two objects away from the CR that named the values. The two quantities are compared as numbers through the chart library's `toFloat`, not as the strings that spell them: `500m` under a ceiling of `1` is the ordinary case for this field, and a lexical comparison refuses it. One thing I deliberately did not guard: a lone request above the ceiling KubeVirt derives is also invalid, but the chart cannot see whether it will be. The derivation only happens in a namespace carrying a ResourceQuota that limits CPU, and the derived value counts IO threads on top of the vCPU count. Failing the render on a value that is perfectly fine in a quota-free namespace would take away a working configuration, so that case is documented rather than enforced. Byte identity for a group that sets neither field is measured rather than asserted: rendering both charts at this branch and at `main` with the knobs unset gives byte-identical `KubevirtMachineTemplate` and `MachineDeployment`, content-hash suffix included. That is load-bearing rather than tidy, a stray key or a moved space renames the template for every node group in every cluster and rolls every live worker VM on upgrade. The existing hash pins in `cluster_test.yaml` cover it too, and stay green. `kubernetes-nodes` gets the field, both guards and its own suite, because that chart is the other half of a byte-identity contract rather than a separate feature surface. Its `tests/render-parity.sh` renders a pool through both charts and compares the four pool objects byte for byte; a field only one side knows would pass that gate and then quietly drop the request when a pool is adopted into a split-out release. The gate gains a case that sets the pair. Two defects in the above turned up while I was checking it, and each is its own commit rather than a quiet amendment. The first is in the chart library: `cozy-lib.resources.toFloat` knew every suffix of the quantity grammar except `u` and `n`, and a suffix it does not recognise is not an error, the unit letter stays on the string, `float64` yields zero silently, and the comparison read zero against zero. A request of `100u` above a limit of `50u` rendered. The library gains the two missing decades and a suite that pins the whole suffix grammar, one value per form, since that failure has no error to notice it by. The second is the value domain, and the third commit settles all of it at once rather than patching whichever edge turned up. Both fields were rendered under `with`, which skips a zero as readily as an absent field, and their guards were keyed the same way; the generated schema behind them admits a leading minus just as readily. So: presence, not truth, is the predicate everywhere either field is read. A negative value is refused, it never reaches a running Pod. A zero limit is allowed and means no ceiling at all rather than the tightest one, because the kubelet turns a zero CPU limit into no CFS quota (`MilliCPUToQuota` returns zero for zero) while the declared key still stops KubeVirt deriving one. A zero request on its own is allowed and reserves nothing. A zero request beside a non-zero limit is refused, and that one is worth stating: KubeVirt's defaulter tests the request with `IsZero`, which an explicit zero satisfies exactly as an absent key does, so it overwrites the zero with the limit and the worker asks the scheduler for its whole ceiling, the outcome `podCpuRequest` exists to prevent, reached silently and reported nowhere. Refusing beats documenting when nothing downstream will ever mention it. A zero limit is the one case that needed a second look, because two reviewers independently read it as "no ceiling, so a positive request beside it is fine". It is not: API server validation refuses a request above a limit that *exists*, and zero is a limit that exists, while KubeVirt keeps deriving a request from the vCPU count whenever the VM declares none, the defaulter that would have copied the limit across does not fire, precisely because the limit is zero. So a lone `podCpuLimit: 0` renders a Pod that is rejected on creation, and it is refused here instead. Paired with `podCpuRequest: 0` it is coherent and stays allowed: no ceiling, no reservation. This settles `podCpuLimit` along with its new pair, deliberately: the two are documented as a pair and cannot sanely disagree about what zero means. Worth stating plainly that `podCpuLimit: 0` changes meaning as a result, at `main` it renders nothing, because `with` drops a zero as readily as an absent field, and here it either renders or fails the render depending on what sits beside it. `podCpuLimit` landed on `main` earlier the same day and has never been in a release, so no shipped configuration is affected, but anyone who set it on `main` between then and now should read this paragraph rather than discover it. Tests are helm-unittest in both packages, mirrored case for case, with one per class of the domain above: the request lands, a fractional quantity survives, the pair renders both keys with the ephemeral-storage pair untouched on both sides, a request alone renders no limit beside it, nothing renders when the field is absent, every guard fails on the message that names its reason, an equal request is accepted, a milli request under an integer limit is accepted, a lone zero request renders, a zero request beside a limit is refused, a zero pair is accepted, and a negative value of either field is refused. I ran a mutation ladder over the result: dropping the request render in either chart, flipping the comparison operator, comparing the quantities as strings, dropping either instancetype guard, dropping the field from the child's group dict, reverting the two suffixes, reverting either predicate to truthiness, dropping either negative guard, and letting the zero-request guard stop excluding a zero limit. Every one is caught, each by the case that exists for it, the string comparison by `500m` under a ceiling of `1`, the suffixes by the micro-range pair, the predicate and the domain rules by their own zero and negative cases. The e2e suite gets the headroom experiment back, `podCpuLimit: 3` with `podCpuRequest: 200m`. The request is what KubeVirt derived for these workers before either field existed, so scheduling is unchanged against every historical run and the ceiling is the only variable that moves. It is still an experiment against a measured mechanism, not a claimed fix for #3513: a per-thread capture on a red run showed both vCPU threads of a stuck worker burning flat out at a ceiling equal to their vCPU count while the guest kernel made no progress, which is what a vCPU spinning on a lock looks like when the vCPU holding it is the one preempted out of the quota they share. If the join failure rate does not move, the lever is elsewhere. ### Screenshots Not a UI change. ### Downstream repositories Walked against the diff. One repository is reached, and it is the one #3859 already reached: `terraform-provider-cozystack` models every app field by hand, so `podCpuRequest` in `packages/apps/kubernetes/values.schema.json` and `packages/apps/kubernetes-nodes/values.schema.json` needs the schema, the model and the expand/flatten pair over there. That work is one edit, not two, both fields land in the same model and the same docs, so the pair is added to the open one, cozystack/terraform-provider-cozystack#32, with a comment that also corrects the description of what `podCpuLimit` does, which that issue took from the claim this PR retracts. The website is not reached: its trigger is adding, renaming or removing a package, and the reference page for an existing package is regenerated from its `README.md` by the release bot. The change to `hack/e2e-chainsaw/_lib/run-kubernetes.sh` is a value inside the suite, not a move, a rename or a make target, so `ccp` is untouched. The remaining repositories restate nothing this diff touches. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: https://github.com/cozystack/terraform-provider-cozystack/issues/32 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(kubernetes): a node group can now set `podCpuRequest`, the CPU request of its worker VMs, next to `podCpuLimit`. Without it KubeVirt copies a declared CPU limit into the request, so raising a worker's CPU ceiling also raised what it asked the scheduler for and could leave the group unschedulable. Unset by default, so existing clusters render and behave exactly as before. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added `podCpuRequest` configuration for independently controlling Kubernetes scheduler CPU reservations. * CPU requests and limits now support explicit zero and fractional quantities. * Added validation for negative values, incompatible instance-type settings, invalid zero combinations, and requests exceeding limits. * Improved resource quantity conversion, including micro- and nano-unit support. * **Documentation** * Updated configuration references and schemas with CPU request, limit, validation, and rollout behavior. * **Tests** * Added comprehensive coverage for CPU resource rendering, validation, quantity conversion, and parity. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Cozystack
Cozystack is a free platform and framework for building clouds.
Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.