mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
0d5b2e35c4f9087ea11aa254add123874446bb4a
7344
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
0d5b2e35c4 |
docs(operator): align scan comments with withholding
Replace three stale descriptions of the removed ceiling-raise behavior with the current withhold and withdraw semantics. Historical explanations of why the raise design was rejected remain intact. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
b3168a3824 |
test(operator): pin system memory defaults
Keep the parser inputs tied to the constants registered as flag defaults, but compare the results with literal 32Gi and 32Mi expectations. A shipped default can no longer move while the defaults case stays green. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
6a521f0c68 |
fix(operator): sweep memory defaults when disabled
List LimitRanges by the ownership label once per Package reconcile when the feature is disabled, and remove managed objects even in namespaces outside the current Package target set. Skip the per-namespace path while disabled and keep cleanup failures non-fatal to namespace reconciliation. Update the flag help, installer guidance, and operations documentation to describe the global cleanup boundary. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
492c9acdd0 |
fix(operator): ignore failed pods from finished jobs
Record terminal Job UIDs while scanning templates, then ignore only Failed pods owned by those Jobs. Failures from unfinished controllers and ownerless pods remain evidence because they may still be recreated. Also remove the stale constructor comment that still described raising a namespace ceiling. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
2e3c40b125 |
fix(operator): ignore non-pod LimitRange scopes
Kubernetes accepts memory keys on PVC-scoped LimitRange items when a storage bound is also present, but those quantities cannot affect pod admission. Restrict the foreign-policy guard to Container and Pod items. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
f6dcb69b61 |
refactor(operator): withhold instead of raising, and drop the grace period
The raise-and-grace machinery cannot work: LimitRanger writes the raised limit onto the very container that justified the raise, so at that container's next admission it carries a limit, the scan skips containers that have one, and the raise has erased its own evidence. The ceiling then drops on schedule and the workload is rejected permanently. Any such machinery approximates admission logic ahead of admission, and that is incomplete by construction: a LimitRange default is one number for a whole namespace and cannot mean "the configured limit, or this container's request, whichever is larger". Only per-pod conditional defaulting expresses that, which is a mutating webhook and not this. The justification for the machinery had also expired. It was designed when the default was 4Gi, where withholding was self-perpetuating for VPA-managed namespaces. At 32Gi nothing in the tree approaches the ceiling -- the largest static memory request is 2Gi -- and VPA sets requests and limits together, so those containers carry their own limit and were never blockers. So a namespace holding a container that requests more than the configured limit with no limit of its own is now left with no default at all, and anything written there earlier is withdrawn. It fails in the safe direction, which is the whole argument: a namespace with no memory.max is a namespace behaving as it did in every release before this, while a default below a request is a namespace whose workload cannot be admitted. It needs no stored state and no timer, because withholding is not a decision that has to survive between reconciles -- once the oversized request is gone the default is applied again on the next pass. Gone with it: the grace period, the clear-since annotation and its clamp, the injected clock, containerMemoryDefault, raisedCeiling, and fourteen tests. The scan stays, because deciding whether to apply or withhold is what it was always for. The foreign-LimitRange ownership guard, the disable path and the 32Gi default are untouched. The residual sharp edge is stated rather than machinery: a workload whose only evidence is a live Pod, built from a custom resource with no template this scan can read, can still be missed if the scan runs while that Pod is absent, and applying the default then rejects its recreation. At 32Gi that is unreachable in this tree. It becomes reachable if an operator lowers the knob below such a workload's request, which the flag help, values.yaml and the operations page now say plainly. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
d3df7c3d4d |
style(operator): gofmt stray blank lines in the package reconciler
Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
588ecbaeeb |
fix(operator): scan pods every reconcile, and key ownership on the label
The pod half of the scan was skipped once a namespace defaulted container memory at exactly the configured limit, on the argument that the shape the scan looks for could no longer be admitted there. That argument proves such a pod cannot come to exist. It does not prove one cannot be attempted, and a failed attempt leaves nothing behind: a workload built from a custom resource whose request grows above the limit after its namespace settled was rejected permanently, with no artefact anywhere that could justify raising the ceiling again. The premise was not sound even for existing pods, since LimitRanger reads LimitRanges through an informer and the shape is admissible in the window before the plugin sees the apply. The skip bought one namespaced List out of six and paid for it with a silent permanent-rejection mode, so it is gone along with memoryDefaultSettledAt and the two tests that pinned it. One of those tests carried the counterexample in its own fixture: a settled 4Gi LimitRange beside an 8Gi pod, asserted invisible. The coexistence guard identified its own object by name. The name is not reserved, so an administrator's LimitRange called cozystack-system-defaults was skipped by the guard and then overwritten by the forced apply, and because LimitRangeSpec.Limits is an atomic list that replaced every item in it, taking their max and cpu policy along. Ownership is now the managed-by label, which is what the delete path has always used. That leaves a same-named object with no label ambiguous: ours with the label stripped, or an administrator's. The two mistakes are not symmetric, so the ambiguous object is treated as foreign and left entirely alone rather than adopted. Re-stamping it with the label goes, and its test is replaced, because the same path reaches an administrator's object and adopting it means the disable path later deletes their policy. Declining to delete is no longer silent either. Also: containerMemoryDefault now takes the last matching item rather than the first, which is what LimitRanger applies, so a multi-item spec no longer reports a ceiling the cluster is not using. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
c6342bf885 |
fix(operator): bring the flag help in line with the guard and removal scope
The --help text still carried two claims that values.yaml and the operations page had already narrowed, and it is the one surface an operator reads without opening the tree. Both were wrong in the direction that costs the reader something. "A namespace where another LimitRange already defaults or caps container memory is left alone" described the guard before it was widened, so an administrator holding a container memory min, a maxLimitRequestRatio or any Pod-scoped bound would read it, expect the LimitRange to be applied there anyway, and get a namespace with no memory.max for a reason the help says does not apply. "Removes any previously created one" is the universal removal claim, and removal is scoped to namespaces an active Package still targets. Two comments corrected while here. constrainsMemory said it switched on the item's own maps "rather than a hand-picked few", which contradicts the hand-picked list immediately below it; what is actually true is that a field added upstream has to be added there too, which is the reason it has its own table test. And memoryDefaultSettledAt claimed the count of raised namespaces falls to none once the oversized requests are gone, which stopped being true when the scan started keeping Failed pods: a carcass Kubernetes never collects holds both the raise and that namespace's per-reconcile pod List open. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
81c12bc953 |
fix(operator): withhold on any foreign memory policy, not two fields
The coexistence guard treated a namespace as already-governed only when another LimitRange carried a container memory default or max, and said so in its comment. That is not the set of fields that collide with what this reconciler writes. A container min above the default request rejects the pod, and so does a maxLimitRequestRatio, since a 32Gi default against a 32Mi default request is a ratio of 1024:1 and no cap allows it. So the namespace-wide admission failure the guard exists to prevent stayed reachable through fields the guard did not read, and kube-system is where it is likeliest, this feature having deliberately brought it in scope. Enumerating the colliding fields does not fix it, because at the Pod scope the collision is not decidable here. Whether a per-container default of 32Gi breaches a pod-wide max depends on how many containers the pod has, and the pods in question do not exist yet: any arithmetic about them is a guess, and a wrong guess costs a namespace its entire pod admission. So the rule is ownership rather than arithmetic. If anything else in the namespace has an opinion about memory, at either scope and in any field, the namespace is not this reconciler's to default. Narrow still on the resource, because nothing here writes cpu or storage and a LimitRange bounding those must keep coexisting. This is the rule the tenant-namespace exclusion already rests on. The pod scan no longer skips Failed pods, only Succeeded. A pod that ran to completion is never recreated from its spec, but an eviction or a node crash leaves a Failed carcass holding the same oversized request that its controller will recreate, and that carcass is frequently the only surviving evidence the namespace needs a raised ceiling, since the shape the scan looks for cannot be admitted once the namespace has settled. Dropping it lowered the ceiling under a workload that was coming back, which is what the grace period exists to prevent. A grace stamp dated ahead of the clock is clamped to now. Unclamped it held the ceiling for the skew plus the grace period, silently, so an hour of skew bought ninety minutes instead of thirty. Clamping never shortens the wait, it only stops it being extended. The two flag defaults are named constants the flag registration and its test both read. The test previously duplicated the literal and still asserted 4Gi after the default moved to 32Gi, so it misinformed a reader about the shipped value and pinned nothing. The disable claim is narrowed to what the code guarantees. "Removes any previously created one" was universal, but the reconciler only runs for namespaces an active Package still targets, so a namespace whose Package was removed keeps its LimitRange. The flag help, values.yaml and the operations page now say that, and name the managed-by label for finding leftovers by hand. The test asserting a Pod-scoped memory max should be ignored is gone, since under the ownership rule a Pod-scoped memory bound withholds. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
df889a6419 |
docs(platform): state when the ceiling drop actually lands
Two imprecisions in the page written alongside the grace period. It said a lowered limit reaches previously-raised namespaces "within" the grace period, which is backwards: the grace period is a floor, not a ceiling, and since reconciles are event-driven the drop lands on the first one after it elapses. And the withdraw path was described only as leaving the namespace alone, without saying that it hands the namespace to the administrator's policy in full, including any admission rejection that policy implies -- which is theirs, and would have happened without Cozystack, but is worth naming where an operator is being told the operator stopped writing. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
30ee57f892 |
fix(operator): route every downward ceiling move through the grace period
The grace period guarded only the branch where the scan came back empty, which left the same defect standing in the branch where it finds something smaller than the ceiling already in place. A namespace raised to 16Gi for a pod visible only as a live Pod, with a second oversized pod at 8Gi beside it, loses the larger pod to a node drain. The scan still finds the 8Gi one, so the empty-scan guard never engages, and the ceiling was written straight down to 8Gi -- rejecting the larger pod's recreation exactly as writing the configured limit would have. A blocker smaller than the standing ceiling is not evidence that a larger one is gone. So the rule is now about direction rather than about why: the ceiling the evidence justifies is computed first, and any move that lowers it goes through the grace period, whichever of the three reasons prompted it -- an empty scan, a smaller blocker, or the configured limit itself being lowered beneath a standing ceiling. The eventual drop lands on what the scan justifies rather than on the configured limit, so a smaller blocker that is genuinely still there keeps the ceiling it needs. The raise is also no longer logged when it changes nothing. A namespace holding a permanently oversized pod re-justifies the same ceiling on every reconcile, and announcing that put a line in the log for every such namespace on every HelmRelease status change. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
769bce59a2 |
docs(platform): say that lowering the knob waits out the grace period too
The grace period was documented as a property of a raised ceiling coming back down, which is where it started, but it applies to any namespace defaulting more than the configured limit -- and a namespace sitting at a previously higher configured value is indistinguishable, on the object, from one the operator raised. So lowering the knob reaches those namespaces within the grace period rather than on the next reconcile, which is worth stating before an operator concludes the setting did not take. The page now also shows how to read the countdown off the objects. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
9fe93a3d7e |
docs(operator): fix three claims that overstated what the code does
The static-pod note said containers built from an on-disk manifest are unaffected "in both directions". True of the container, which the kubelet runs from disk whatever the API says, but the mirror pod is an ordinary API object that passes admission like any other, so a static pod declaring a memory request above the ceiling with no limit of its own has its mirror pod rejected -- leaving the container running with no representation in the API. Worth naming, since kube-system is in scope by design. The scan does cover it: a mirror pod is a live Pod. The ReplicaSet exclusion was justified on live ones being copies of their Deployment's template. That does not reach an ownerless ReplicaSet, which at replicas 0 has neither pods to be seen through nor a Deployment template standing in for it. It stays out of scope -- nothing here ships a bare ReplicaSet -- but as a gap left open rather than one the stated reason closes. The delete guard's comment claimed the enabled apply always adopts and stamps a same-named foreign object. It does not when that object's spec already matches, because then the apply is skipped entirely. The outcome is the same one the guard gives on purpose, but it is reached by the apply not running. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
fe792369e5 |
docs(platform): bring the memory-limit page in line with the operator
docs/operations/system-memory-limits.md merged ahead of the code it describes, so main documents two installer knobs and a 4Gi default that exist nowhere outside this branch. An operator following the shipped page today sets a value that silently does nothing. The page is corrected to the default this branch ships and gains the three behaviours it had no account of at all: that a namespace holding a request above the ceiling is raised rather than skipped, that a raised ceiling only comes back down after the scan has been continuously empty for a grace period and why that delay has to exist, and that a namespace another LimitRange already governs is left alone. Each names the log line to grep for, since none of the three is visible in the object. The ceiling rationale is also stated more honestly: the scan reads declared requests and never a container's real resident set, which is the reason the default sits far above any plausible working set rather than close to one. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
1df6e3ac8f |
revert(linstor): leave the controller memory limit to its own PR
This branch gave linstor-controller a 10Gi memory limit, justified in the values and pinned in cluster_test.yaml on the grounds that the namespace default was 4Gi, half the -Xmx8G baked into the image. Both halves of that have moved. The default is now 32Gi, four times the heap ceiling rather than half of it, so nothing about the JVM requires a per-chart limit to sit above the namespace default any more. And #3824 argues the opposite direction for the same component -- extending the controller's existing limit-free exemption to linstor-satellite, using that exemption as its precedent. Keeping the 10Gi here would remove the precedent that PR reasons from, and whichever of the two merged second would silently invalidate the other's rationale. So linstor policy belongs to one PR, and it is not this one. The three files go back to their state on main; this branch is now confined to the operator and the installer chart. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
e131a49514 |
feat(operator): default the system namespace memory ceiling to 32Gi
The guard in front of this default reads declared memory requests. It cannot see a container's real working set, so a system component that requests little but legitimately uses more than the ceiling, and declares no limit of its own, was handed a deterministic kernel OOM on its next recreation in exchange for leaving the Talos handler's victim set. Since the feature ships enabled, that trade fell on the whole fleet on upgrade. It is not a trade this feature has to make. The handler discards a cgroup for carrying a memory.max at all, whatever its value, so immunity does not require a fitted ceiling -- only a real one. A default far above any plausible working set buys the immunity and leaves nothing for the ceiling to bind on, which also keeps it clear of the failure mode this repository already carries a scar from, where an undersized limit converted a rare pressure-driven kill into a reliable one. Nothing about scheduling changes: only requests are scheduled against, and the paired default request stays 32Mi. An operator who wants the ceiling to bind lowers it, per component in a chart or fleet-wide with this knob, once real usage is measured -- which is the direction that needs evidence, and now the direction that needs it rather than the default. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
ad3c9ef04e |
fix(operator): stay out of a namespace another LimitRange governs
The reconcile only read its own LimitRange by name, so a system namespace already carrying an administrator's LimitRange got a second one beside it. LimitRanger applies defaults from every LimitRange in the namespace and then validates the pod against every one, and nothing orders them, so the effective container ceiling moved between the administrator's value and this one for reasons no chart author could see. That is the same hazard this feature cites to keep itself out of tenant namespaces, and it does not stop being one because the namespace is a system namespace. A max is the sharper half. This reconciler writes a default and no max, so an administrator's LimitRange carrying a container memory max below the configured default had every container that declares no limit of its own rejected: the default is written first, then validated against the max it exceeds. That takes out a whole namespace, and it reaches kube-system, which this feature enters for cozystack-scheduler and which is where a distribution's or an administrator's own LimitRange is likeliest to sit. Such a namespace is now left alone and logged, and anything this reconciler wrote there earlier is withdrawn rather than left sitting beside the foreign object, since that object is half of the ambiguity. The check is deliberately narrow -- only a container memory default or max collides with what is written here -- because read more widely it would be a blanket opt-out that an unrelated cpu policy could trip. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
5630954e34 |
fix(operator): hold a raised memory ceiling before lowering it
A raised container memory ceiling was recomputed from the scan on every reconcile, so a single scan that found nothing lowered it back to the configured limit. An empty scan is not evidence that the oversized request is gone: it is equally consistent with the pod being between deletion and recreation. For a blocker that appears only as a live Pod, which is a class this repo ships -- every CNPG Cluster in a system namespace builds its instance pods directly rather than through a StatefulSet -- an ordinary Talos node drain opens that window, and reconciles follow HelmRelease status churn, of which a node upgrade produces plenty. The ceiling was lowered under a pod that was merely being rescheduled; its recreation was then defaulted a limit below its own request and rejected at admission for good, because the rejected pod never exists so no later scan can find it, and the namespace settles at the configured limit where the pod half of the scan stops running at all. The drop was not even logged. Lowering now waits until the same emptiness has held for memoryCeilingDropGrace, with the instant it was first observed recorded on the object so it survives an operator restart and a change of leader. A wall clock rather than a count of consecutive clean scans, because the reconcile rate rises exactly when the risk does and any small count can be burned inside one rescheduling window. A blocker seen again cancels the countdown, an unreadable stamp restarts it, and both the hold and the drop are logged. The steady-state comparison now also covers the two metadata fields this reconciler manages, because the grace stamp is one of them. That incidentally fixes the managed-by label never being re-stamped when only metadata had drifted, which had left turning the knob off unable to remove an object the operator did own. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
b6cb0def78 |
docs(operator): say that kube-system is in scope for the memory default
The scope of the default LimitRange is every namespace a Package targets whose name does not start with `tenant-`, and cozystack-scheduler installs into kube-system, so a namespace owned by the underlying Kubernetes distribution is defaulted like any other. Neither the flag help nor the installer value said so, and both are what somebody setting the knob reads. docs/operations/system-memory-limits.md already covers this, including the part that surprises: containers the kubelet builds from an on-disk manifest never pass through admission, so a Talos control-plane static pod gets no memory.max from this feature in the first place and the mirror pod is the only thing the LimitRange reaches. Both notes now carry the same statement, so neither reading of the knob leaves it out. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
3c1d8d6988 |
docs(operator): bound the managed-by guard to the disabled state
The comment on the label check in deleteSystemDefaultsLimitRange reads as a general promise not to touch an administrator's own LimitRange of that name. It is narrower than that, and the difference is worth writing down: while the feature is enabled the apply takes the name over with ForceOwnership and stamps the managed-by label itself, so a hand-written cozystack-system-defaults is adopted and then deleted here when the knob goes off. That is the intended SSA behaviour, not a defect. What the label actually buys is that a LimitRange of that name the reconciler never wrote to survives the knob, which is a real guarantee and a smaller one. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
accb4f9e1b |
refactor(operator): drop the dead error check from the settled-state guard
The guard read `err == nil && apiequality.Semantic.DeepEqual(...)`, which looks like it consults the earlier Get and does not. That Get's error was reassigned by the scan two statements up, and the scan's non-nil case has already returned, so the clause is always true by the time it is evaluated. It was harmless only by accident: a namespace whose Get returned NotFound carries an empty spec, which never compares equal to the desired one, so the apply happened anyway. Removing the clause makes that the reason rather than a coincidence sitting behind a condition that cannot fire. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
ead330ccc3 |
fix(operator): keep scanning pods while the ceiling stays raised
The pod half of the memory scan was skipped whenever any container memory default was in force. That is the weaker of the two available conditions, and it silently retracted a ceiling this reconciler had just raised. The default written into a system namespace is either the configured limit or a ceiling raised above it to clear an oversized request, and the raise is recomputed from the scan every reconcile rather than read back off the object. So a raise that only the pod half could justify -- a live pod requesting more than the limit with no limit of its own, in no workload template the scan can read, which is what an operator building pods from its own custom resource produces -- lasted exactly one reconcile. The raised LimitRange satisfied "in force", the next reconcile skipped the pod list, the scan came back empty, and the configured limit was reapplied underneath a still-running pod. That pod's next recreation was rejected with "must be less than or equal to memory limit", and because the skip stayed in effect nothing ever raised the ceiling again: a healthy component goes down at a node reboot and stays down, with the revert unlogged. Lowering the knob reached the same place with no race in it. An operator running at 16Gi has pods admitted with requests only that ceiling allowed; dropping the flag to 4Gi wrote 4Gi straight over them. Gate the skip on the namespace already defaulting memory at exactly the configured limit instead. That keeps the pod list alive for as long as a namespace sits raised, which is what lets the raise be re-justified or dropped on evidence rather than frozen or forgotten, and it still skips the list in the steady state the skip was added for. The comparison is by value for the same reason the spec comparison is semantic: 4096Mi and 4Gi are one ceiling. The cost is that a raised namespace reads the pod list every reconcile. That is the price of being able to lower the ceiling again, bounded by how many namespaces sit raised -- none, once the oversized requests are gone. Three tests cover it, each of which fails on the old condition: a raised ceiling held while the pod needing it runs, the same ceiling dropping back once that pod is gone, and the configured limit falling below a ceiling already in place. A fourth pins the notation equality. The fixture comment on the steady-state test claimed its 8Gi pod could not exist beside a settled 4Gi LimitRange; it can, through a narrow window, so it now says so. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
26a1ce5be0 |
test(operator): drop the incidental fixture from the no-pods case
The scaled-to-zero test seeded a spec-less LimitRange, which is not part of the case it names. It is the first application in the namespace that makes a workload with no pods invisible to a pod-only scan, so start from no LimitRange at all and let the assertion rest on the workload rather than on the fixture. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
b62fac474c |
refactor(operator): move both server-side applies off the deprecated client.Apply
client.Apply as a patch type is deprecated in controller-runtime v0.23.1 in favour of Client.Apply taking a runtime.ApplyConfiguration. Convert both call sites — the system defaults LimitRange, new on this branch, and the Namespace apply that predates it — keeping the same field owner and force-ownership options. The LimitRange keeps its typed builder, because that is what the spec comparison and the tests read, and gains a small function that restates it as an apply configuration immediately before the wire. The Namespace apply now sends only the labels and annotations the reconciler actually sets, rather than a whole typed object whose spec and status were empty. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
c7af79b734 |
fix(operator): only delete the LimitRange this reconciler owns
deleteSystemDefaultsLimitRange matched on name alone, and the name is not reserved: nothing stops an administrator creating a LimitRange called cozystack-system-defaults by hand. Turning the knob off would then silently remove policy this operator never wrote, in every system namespace at once. Stamp app.kubernetes.io/managed-by on the object the reconciler applies and require it before deleting. Anything else is left where it is, which is also the right answer if the name is ever taken over by another component. Nothing in the wild carries the old unlabelled shape, since the feature has not shipped. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
dbc5e5ea13 |
perf(operator): read before deleting the disabled LimitRange
With the limit set to 0 the disabled path runs for every system namespace on every Package reconcile and issued a DELETE each time, swallowing the 404 that nearly all of them returned. That is a write attempt per namespace per reconcile, forever, and every one of them lands in the audit log. Get from the informer cache first and return when there is nothing there. The test now counts deletes, so a second reconcile over an already-empty namespace has to stay at zero rather than merely not erroring. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
89b78cbf20 |
docs(operator): drop the duplicated field from the ceiling-raise log
request and raisedLimit were logged with the same value, because the raised ceiling is the blocking request. Keep request, which names the thing an operator has to go and change, and say in the message that the ceiling becomes it. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
9866113da0 |
test(operator): pin the system namespace memory flag parsing
parseQuantity and the request-above-limit check were closures inside main(), so no branch of either was reachable from a test: not empty, not malformed, not negative, not a request of 0 against a real limit, and not the pair check itself. That last one is the one worth having. A limit and a request that each parse but contradict each other produce a LimitRange the API server rejects, once per system namespace on every Package reconcile, and nothing in that error points back at a flag. Lift both into parseSystemNamespaceMemory, which returns an error instead of calling os.Exit, and pin all of it: the shipped defaults, empty as the off switch, an explicit 0 limit beside a set request, a 0 request against a real limit (supported on purpose), malformed and negative values on either flag, request above limit, and request equal to limit as the legal boundary. main() keeps the same exit-on-error behaviour, now in one place. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
dd36eeddbc |
docs(operator): describe the raise, not the retract, in the shipped help
The flag help and the installer values comment both still said the operator skips a namespace holding a request above the configured limit and retracts any LimitRange it applied there. It stopped doing that in 3127c0fad: it raises that namespace's ceiling to clear the request and drops back once the request is gone. An operator reading the shipped help and then finding an 8Gi LimitRange in cozy-monitoring had no way to connect the two. Both texts now describe the raise, the drop-back and the log line to grep for. The values comment also loses its VPA warning, which this branch's own comment in findRequestAboveDefaultLimit refutes — VPA rescales the limit in proportion to the request under the default controlledValues, so it cannot be what puts a namespace over — and gains a pointer to declaring a limit in the component's own chart, which is how packages/system/linstor now stays out of the default's reach. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
ae431808ea |
fix(linstor): give linstor-controller a memory limit above its baked-in -Xmx8G
linstor-server bakes `-Xms256M -Xmx8G -XX:+CrashOnOutOfMemoryError` into the generated bin/Controller (build.gradle, jvmControllerOpts), nothing in this chart or in piraeus-entry.sh overrides it, and a JVM handed an explicit -Xmx never reads memory.max. The container was deliberately left with a 700Mi request and no limit at all, which was the safe choice while nothing else supplied one. It stops being safe alongside the system-namespace LimitRange: a container declaring no memory limit is given the configured default, 4Gi, so the process's own heap ceiling would sit at twice its cgroup ceiling and a large enough DRBD resource count would have the kernel kill the storage control plane where the JVM would otherwise have crashed itself loudly at 8G. Stating the limit here is also what keeps the container out of the LimitRange's reach, since a container with its own limit is never defaulted one. 10Gi is the 8G heap plus room for metaspace, code cache, thread stacks and direct buffers. CPU stays unlimited. The helm-unittest assertion that pinned "no limits" is replaced by one pinning the memory limit and the absence of a CPU limit, so neither lowering it below the heap ceiling nor dropping it again passes silently. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
87f79c17c3 |
fix(operator): keep scanning templates after the LimitRange settles
The steady-state fast path returned as soon as the LimitRange matched the configured spec, on the grounds that LimitRanger gates anything a scan could still find. That holds for pods and not for templates: a Deployment added to a settled namespace with a memory request above the ceiling and no limit of its own is admitted exactly as written, and only the pods it then tries to create are rejected with "must be less than or equal to memory limit". Nothing raised the ceiling, because every later reconcile returned at the same line, so a chart bump became a rollout that could never produce a pod and logged nothing about it. Split the scan instead. Templates are read on every reconcile; the pod list, which is the expensive half, is read only while no memory default is in force in the namespace — once one is, LimitRanger fills in a limit for every container admitted without one, so a container carrying a request with no limit beside it can no longer come into existence. The skip keys on the default actually being in force rather than on the object existing, so a LimitRange whose spec was stripped scans pods again. The write is now guarded by a semantic comparison against the spec the scan settled on rather than against the configured spec, so a namespace already sitting at a raised ceiling stops taking an identical apply on every reconcile as well. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
c9cacda1a0 |
docs(operator): pin the admission order by rule, not plugin index
The scan's justification for treating a VerticalPodAutoscaler as harmless rested on LimitRanger being "position 6 of AllOrderedPlugins" and MutatingAdmissionWebhook being "35". Both numbers are properties of one release rather than of the ordering, and they had already drifted out from under the comment. Under k8s 1.35, which this tree builds against, LimitRanger is the 6th entry but MutatingAdmissionWebhook is the 33rd; index 35 is ResourceQuota and the 35th entry is ValidatingAdmissionWebhook, neither of which mutates, so the sentence named the wrong plugin under either counting convention. In 1.31 the same plugin sat at 30, and SecurityContextDeny being dropped in 1.35 is what moved LimitRanger. Restate it on the invariant the list asserts about itself -- webhook, resourcequota and deny plugins must go at the end -- with LimitRanger compiled in above that marker and every plugin that can rewrite a container's resources below it. That survives a release bump where an index does not, and it covers MutatingAdmissionPolicy as well, a second mutator that the positional wording predated and never accounted for. Record what the order buys, which the old wording left implied. Run the other way round, a plugin mutating before LimitRanger would write a request into a container that has no limit yet, LimitRanger would then default the limit to the configured ceiling, and core validation would reject the fully mutated pod. The scan can see none of that -- the request is in no template and the pod is never admitted -- so the order is the only thing keeping VPA out of the residual gap this comment already records for controlledValues: RequestsOnly. Fix the matching rationale on the live-pod case in TestFindRequestAboveDefaultLimit, which still claimed the pod's request was raised by a webhook after the template was rendered. The ordering rules that out; what the case actually rests on is a pod admitted before the LimitRange existed, or one its controller builds from a custom resource no template in the scan can show. No behaviour change, and no new test: a container that carries its own memory limit is already pinned out of the scan's scope for a template, in TestFindRequestAboveDefaultLimit, and for a live pod, in TestReconcileSystemDefaultsLimitRangeAppliesWhenLargeRequestCarriesItsOwnLimit. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
7eecbe9b4c |
docs(operator): document a zero default memory request
--system-namespace-memory-request=0 passes the startup check, since 0 is not greater than the limit, and produces a LimitRange with defaultRequest: 0. Whether that is a footgun to reject or a supported setting to explain was open, so I asked a real API server: envtest with kube-apiserver 1.36.0 and LimitRanger enabled, with a second namespace using 32Mi as the positive control that the plugin was actually running. It is supported. The API server accepts the LimitRange and stores defaultRequest as 0 rather than folding it back to the default -- the Default-to-DefaultRequest copy only fills keys that are absent, and this one is present. A container declaring no memory of its own is then admitted with an explicit request of 0 and a limit of 4Gi, QoS Burstable. So LimitRanger does not read zero as absent, and the container still gets memory.max on its cgroup, which is the whole reason this feature exists. What it costs is the scheduling signal: nothing is reserved for that container, which is exactly where system components stood before this feature. Say so in the flag help text, where an operator setting the knob will see it, and note at the startup check that 0 passing is deliberate so nobody hardens it into a rejection later. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
16b8ea4f0e |
docs(operator): correct the VPA rationale on the request scan
The comment argued that the danger this scan guards against is a VPA webhook writing a request above the defaulted limit, which is then rejected at admission. Admission ordering rules that out. LimitRanger is position 6 of AllOrderedPlugins and MutatingAdmissionWebhook is 35, so the LimitRange has already defaulted the container's memory limit before VPA is called, and VPA rescales that limit in proportion to the request it writes (GetProportionalLimit, under the default controlledValues: RequestsAndLimits). Request and limit stay consistent and the pod is admitted. Rewrite it around what can actually happen: a request already in the spec when it reaches admission with no limit beside it. In this tree that means an operator lowering --system-namespace-memory-limit below a statically declared request -- the largest is 2Gi, in packages/system/rabbitmq-operator, so the 4Gi default clears them all today. Two narrower shapes are real too: a pod that predates the LimitRange, whose own controller may build it from a custom resource no workload template in this scan can show; and a VPA set to controlledValues: RequestsOnly, where nothing rescales the defaulted limit and the rejection returns. The second doubles as the residual gap, since a first pod rejected outright appears in neither a template nor a pod list. The scan itself is unchanged; only the justification was wrong. Also record why withholding the LimitRange had to go rather than merely being suboptimal: in a namespace with no LimitRange nothing defaults a memory limit, so the next VPA-sized pod arrives with a large request and no limit, the scan finds it again, and the namespace is withheld again. The old behaviour could never get its foot in the door in cozy-monitoring. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
d682158d1f |
test(operator): pin the tenant namespace boundary
One strings.HasPrefix keeps this feature out of tenant namespaces, and nothing tested it: "tenant-" appeared twice in package_reconciler.go and not once in the test file, so replacing the check with isSystem := true left the whole package green. What it protects is not subtle. packages/apps/tenant ships tenant-range-limits, which defaults container memory to 128Mi in every tenant namespace that declares resourceQuotas. A system LimitRange landing beside it leaves two LimitRanges defaulting the same containers, and which of them wins is decided by the order LimitRanger happens to iterate them -- neither chart controls that -- so a tenant container's default memory limit would move between 128Mi and 4Gi with nothing in the tree changing. Reconcile a variant with one system and one tenant namespace, and assert no LimitRange is written for the tenant one. The system namespace in the same run is the positive control: without it the test would still pass if LimitRanges stopped being written anywhere at all. The cozystack.io/system label is asserted too, since the same boundary gates both. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
2e1ab6682c |
fix(operator): raise the namespace ceiling instead of retracting it
When a container in a system namespace requests more memory than the configured default, the guard used to withhold the LimitRange and delete any it had applied earlier. That leaves the namespace with no default memory limit at all, which is the precise state this feature exists to remove: without memory.max on the pod cgroup the Talos OOM handler keeps the pod in its victim set, and a tenant workload's memory pressure gets paid for by a system component. Default that namespace's memory to the offending request instead. A looser ceiling still sets memory.max, so the pod scores zero in victim selection exactly as it would at the configured limit -- 8Gi protects it as well as 4Gi does, and both protect it infinitely better than nothing. The raise needs no knowledge of where the request came from, and it undoes itself: once the request is gone the scan finds nothing, desired falls back to the configured limit, and the next reconcile applies it. This is not a hypothetical namespace. packages/system/monitoring pins maxAllowed.memory at 8Gi for vmselect and vmstorage, packages/system/monitoring-agents pins 6G for vmagent, all three run updateMode: Initial, and no monitoring template declares a memory limit. Initial never evicts, so those requests do not come down on their own: under the old behaviour cozy-monitoring, which holds the largest memory consumers on the platform, was permanently excluded from the feature on exactly the clusters it was written for. A raised namespace no longer matches the base desired spec, so it scans on every reconcile. That is deliberate: the raise has to track the current largest request in order to be able to drop back, and caching it would freeze a ceiling that nothing would ever lower again. The set of raised namespaces is small. The log moves from Error to Info. Nothing is wrong when this fires; the operator is reporting a ceiling it chose. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
c65db1ad52 |
perf(operator): skip the memory scan when the LimitRange is settled
reconcileSystemDefaultsLimitRange runs for every system namespace on every Package reconcile, and Package reconciles follow HelmRelease status churn through Owns, so on a platform with dozens of system namespaces the pod and workload scan behind the guard is by far the most expensive thing in the function -- and it is uncached by design, because caching it would mean a cluster-wide Pod informer in an operator whose job here is to bound memory. Build the desired LimitRange first, read the existing one from the informer cache, and return without scanning when the two specs already match. That is safe because the LimitRange being applied is itself the guarantee: from that moment LimitRanger rejects any new pod whose memory request exceeds this default, so the only requests a scan could still find are ones that predate the LimitRange, on pods already running, and there is nothing it would do differently about them. Steady state becomes one cached GET. The comparison is apiequality.Semantic, not reflect.DeepEqual, because resource.Quantity carries its parsed format and cached string: 4Gi and 4294967296 are the same ceiling and must not read as drift, or every namespace would scan and re-apply forever. The test counts List calls rather than only checking the resulting LimitRange, which cannot tell a skipped scan from a repeated one that reached the same answer, and seeds an 8Gi pod so a scan that ran would also change the object. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
b55c4f347a |
docs(operator): keep the residual-gap note on the request scan
The pod-only scan documented what it could not see: a request that only ever exists post-admission, on a workload whose first pod is rejected outright. Reading pod templates narrows that gap rather than closing it, and the rewritten comment dropped the note entirely, which reads as a claim that nothing is left. Restore it in its narrower form: a template under the limit, a webhook that raises the request past it, and a first pod rejected before it exists, is still invisible to this scan. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
10acfe025d |
test(operator): cover both terminal pod phases in the skip
findRequestAboveDefaultLimit skips a pod in PodSucceeded or PodFailed, because neither is ever recreated from that spec. Only the Succeeded half had a fixture, so deleting the PodFailed clause left the whole ./internal suite green -- half the condition was uncovered. Give the case a Failed pod as well. Deleting either clause now fails it. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
654412acd2 |
fix(operator): scan workload templates for oversized memory requests
reconcileSystemDefaultsLimitRange withholds the LimitRange when a container in the namespace requests more memory than the configured default limit, because LimitRanger would then default a limit below that request and the API server rejects the pod. The scan behind that decision listed pods only, so it could not see a workload that currently has no pods: a Deployment or StatefulSet at replicas: 0, a CronJob between runs, a Job with spec.suspend. Those are ordinary steady states rather than corner cases. The namespace looked safe, the LimitRange was applied, and the pod was rejected at the next scale-up or the next schedule with "must be less than or equal to memory limit of 4Gi" -- a workload that cannot come back, discovered at the moment somebody needs it. Scan the pod templates of Deployments, StatefulSets, DaemonSets, CronJobs and Jobs as well as live pods. Both halves are needed: a template is the only place a scaled-to-zero workload appears at all, and a live pod is the only place a request that the VPA admission webhook raised after admission appears. ReplicaSets are deliberately left out -- a live one copies its Deployment's template, and the older revisions a Deployment keeps carry superseded templates that only an explicit rollout undo would ever instantiate, so scanning them would withhold the LimitRange over a request replaced releases ago. Jobs are named separately because not every Job has an owner whose template would cover it, and a finished one is skipped for the same reason a terminated pod is. A suspended CronJob is not skipped: suspend is the same dormancy as replicas: 0, undone by the same kind of human action, and exempting it would leave the gap open for one kind. Templates are scanned before pods and the comparison stays strict, so a workload and its own pods report once, as the workload -- the object an operator has to edit, and one that outlives pod names. A live pod outranks its own template only when its request is strictly larger, which is exactly the webhook-injected case. memoryRequestBlocker therefore carries Kind/name instead of a bare pod name, and the log key is "workload" rather than "pod". Every added List fails closed, which the caller already relies on: a scan that cannot read is not evidence that applying is safe, so the namespace is skipped and logged instead. The test for that seeds a sentinel LimitRange, because asserting the object still exists cannot tell "left alone" from "applied blind" -- a swallowed List error applies a fresh LimitRange that exists just the same. RBAC needs no chart change: the operator's ServiceAccount is bound to cluster-admin, so it can already list these kinds. The kubebuilder markers are extended anyway to record what the controller reads. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
7cc090b05f |
test(operator): pin the largest-offender scan to all three fields
The "largest offender is reported" case asserted only the container name, and systemPod named every container "app", so the assertion held whichever of the two pods findRequestAboveDefaultLimit had picked. The comparison that keeps the largest request was uncovered. Name the fixture container after its pod, assert the reported pod, container and quantity together, and give the case three offenders whose largest sits in the middle of the name-ordered pod listing, so keeping the first, the last or the smallest offender all name a different pod and quantity. The init-container case now pins the same three fields. No production behaviour changes. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
546d8e6a54 |
feat(operator): default system namespace memory via LimitRange
The Talos userspace OOM handler (v1.12+) picks its victim by ranking
cgroups with `memory_max.hasValue() ? 0.0 : {Besteffort: 1.0, Burstable:
0.5, ...}[class] * memory_current`, discarding every cgroup that scores
zero. Only a memory limit sets memory.max on the pod cgroup, so only a
limit takes a pod out of the candidate set: a request merely demotes
BestEffort to Burstable, which under the v1.13.6 default
strictCgroupClassOrdering still means "killed second". PriorityClass has
no effect at all, because the handler walks cgroupfs and never talks to
the API server.
Victim selection is decoupled from the trigger, so the cgroup that
caused the pressure and the cgroup that gets SIGKILLed are unrelated by
design. Tenant workloads generally carry limits and system components
generally did not, which inverted the intended order: a tenant workload
thrashing against its own limit drives node-wide memory PSI, and
metallb-speaker or linstor-satellite is killed for it while the workload
responsible is never touched.
An audit found roughly 70 workloads across ~30 packages with at least
one container missing a memory limit, so per-chart resources would
neither cover the current set nor keep covering it. Maintain a
LimitRange in every system namespace instead, at the single choke point
that already creates them and already distinguishes system from tenant
(reconcileNamespaces). Tenant namespaces are left alone because
packages/apps/tenant owns LimitRange policy there.
The limit is a ceiling, not a reservation: it defaults well above real
usage (4Gi against a largest observed system request of 512Mi).
defaultRequest is set explicitly and small, because Kubernetes otherwise
defaults each request to the limit and reserves that much at schedule
time. Setting the limit to 0 disables the LimitRange and removes any the
operator previously created.
Two things keep the feature from being able to hurt more than it helps.
A LimitRange default only reaches a container that declares no memory
limit of its own, and the API server then rejects a request above its
limit. So before applying, scan the namespace for a container that
requests more than the configured limit and sets none, and where one
exists skip the namespace, retract any LimitRange applied earlier, and
log the pod, container, request and limit. This is not hypothetical:
packages/system/monitoring and packages/system/monitoring-agents ship
VPAs for cozy-monitoring with maxAllowed.memory of 8Gi and 6G under
updateMode: Initial, so the VPA admission webhook -- which runs after
the in-tree LimitRanger -- can write a request past 4Gi that no chart in
the tree declares. That is why the check runs against live pods rather
than rendered templates, and why it reads through APIReader rather than
starting a cluster-wide Pod informer.
And no failure inside the feature propagates: a LimitRange error, or a
pod list that cannot be read, is logged and stepped over instead of
failing the Package reconcile. A namespace without a LimitRange is back
to the behaviour of every earlier release, while a namespace that never
reconciles leaves its components uninstalled.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
|
d2e3eb4f82 |
fix(kubernetes, e2e): unblock autoscaling below min size, size the computeplane pool, fix the backupcontroller build (#4768)
## What this PR does Makes the backupcontroller test package compile again on main. #4587 added a flavor argument to `createCNPGBackupArtifact` and `marshalCNPGBackupSnapshot`, and #4669, whose CI ran before that, added two test calls with the old argument lists. Both PRs were green on their own, and together they break `go test ./internal/backupcontroller/`. The two calls now pass the same default flavor as the other tests in the file. The second commit fixes an e2e suite that fails the required E2E check here. It is the `computeplane` suite from #4713, and its worker pool was one node with a single vCPU. After the kubelet reservations that node has 900m of CPU to allocate. The tenant addons (cilium, coredns, csi, ingress-nginx, metrics-server, cert-manager) request exactly 900m between them. If cert-manager's startupapicheck hook pod is created last, it can't be scheduled and the install times out after 10m. The reinstall after remediation runs into the same full node. Whether that pod comes last depends on install order, so the suite passed on #4713 and failed here. The pool now has two workers with one vCPU each, the same shape the kubernetes suites use. One vCPU per worker stays on purpose, because of #3513. Two workers exposed a deadlock in the tenant cluster-autoscaler. The suite's pool is created at zero replicas and then gets a min size of 2, which CAPI does not apply to the live replica count on upgrade (#4786). The autoscaler's scale-up for a pending pod asks for one node, and the clusterapi provider refuses it because one is under the min size, although it is an increase. That refusal backs the group off, and the min-size enforcement skips groups in backoff, so the pool never leaves zero. Upstream had the same report in kubernetes/autoscaler#4942, closed as stale, and it is now tracked in kubernetes/autoscaler#10400. Our autoscaler image already patches the mirror case for max size, and the patch now refuses only a decrease below the min size. `go vet` and `go test ./internal/backupcontroller/...` pass, and so do the bats pins for the computeplane suite. The autoscaler patch adds a test for each direction, and the clusterapi package tests pass with it applied to 1.32.0. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved cluster autoscaling for worker groups outside configured size limits: groups below their minimum can scale up toward it, and nodes can be removed when a group exceeds its maximum. Scale-down requests below the minimum remain restricted. * **Tests** * Updated backup and restore test calls with PostgreSQL and empty flavor values. * Updated ComputePlane checks for a two-worker pool and clarified cert-manager installation timing expectations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
82ae9f6cf7 |
fix(kubernetes): let the autoscaler grow a node group that sits below min
The clusterapi provider refuses any SetSize under the group's min size, including an increase. A pool whose min size is raised while it stands at fewer replicas then deadlocks: the scale-up for pending pods asks for fewer nodes than the min, is refused, and puts the group in backoff, and the min-size enforcement skips groups in backoff. The group stays where it is for good. Refuse only a decrease below min size. The guard it keeps is the one DecreaseTargetSize relies on, and DeleteNodes checks min size itself. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
4c3c838a66 |
test(e2e): run the computeplane pool at two workers
A single-vCPU worker has 900m to allocate after the kubelet reservations, and the tenant addons' CPU requests use all of it. When cert-manager's startupapicheck hook pod is the last one created it stays Pending, the install times out, and the reinstall after remediation runs into the same full node. Whether it is last depends on install ordering, so the suite passed or failed on the race. Two workers match the kubernetes suites' pool and leave room for the hook pod. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
9620be0103 |
feat(postgres): add per-user createRole and bypassRls attributes (#3505)
## What this PR does - Adds `users.<name>.createRole` to the postgres chart, which sets the `CREATEROLE` role attribute on a managed login user. - Adds `users.<name>.bypassRls`, which sets `BYPASSRLS` on the same user. The two attributes are needed together to provision a privileged service role, see below. - Sets `createrole_self_grant` as a role-level default alongside `createRole`, without which the attribute is granted but not usable. - Refuses `createRole` on PostgreSQL 15 and older, where it is an escalation path rather than a bounded privilege. - Makes the init job record the roles it manages in a superuser-only marker instead of trusting role comments and refuses to adopt a role it did not create once `createRole` has been used, see below. - Adds `tests/user_createrole_test.yaml`: defaults, both explicit values for each attribute, composition with `replication` and with each other, the `createrole_self_grant` set and reset paths, and the version guard on v15 and v13. Adds `tests/managed_roles_test.yaml` for the marker and the adoption checks. Today a tenant that needs to create its own roles has no way to do it through the chart, because role attributes are not inherited through role membership: granting `CREATEROLE` to the generated `<db>_admin` group role has no effect on the users that are members of it, and `<db>_admin` is `NOINHERIT` anyway. The attribute has to be set on the login user itself, which only the init script can do. The remaining option is to hand the tenant the `postgres` superuser, which also gives it `COPY ... FROM PROGRAM`, arbitrary file access and command execution inside the instance pod. These flags are the narrow version of that: the tenant manages its own roles, and the superuser stays internal. Both attributes are reconciled on every init-script run, like `replication`, so setting one back to `false` emits the `NO...` form and revokes it. ## Why `bypassRls` belongs in the same PR `CREATEROLE` on its own does not let a tenant reproduce a role layout that uses row-level security, which is the common case this feature exists for. PostgreSQL only lets a role create other roles carrying `BYPASSRLS` if it holds the attribute itself, so `CREATE ROLE service_role ... BYPASSRLS` fails with `Only roles with the BYPASSRLS attribute may create roles with the BYPASSRLS attribute`. Exposing `createRole` without `bypassRls` therefore ships a feature that stops one statement short of the goal. `bypassRls` is not gated on a PostgreSQL version, because it carries no escalation of its own: `BYPASSRLS` waives row-level security inside the tenant's own databases, and the database owner already reads every row in them. It grants no reach outside the tenant's data. ## Why `createrole_self_grant` is set with the attribute This is the non-obvious half. PostgreSQL 16 grants a `CREATEROLE` holder only `ADMIN OPTION` on the roles it creates, governed by `createrole_self_grant`, whose default is empty. `ADMIN OPTION` allows granting the role onward but not `SET ROLE` to it, and creating an object owned by another role requires `SET`. The attribute therefore looks functional -- `CREATE ROLE` succeeds -- while the very next statement a tenant writes, `CREATE SCHEMA ... AUTHORIZATION <new role>`, fails with `must be able to SET ROLE`. Every tenant granted `createRole` would hit this and have to discover the parameter, so the chart sets it as a role-level default and resets it when the attribute is withdrawn. ## Why the init job stops trusting role comments and role names The init job runs as the superuser, and it trusted two things a CREATEROLE holder controls for the roles it creates. It picked the users and roles to drop by their comment, and on PostgreSQL 16 a CREATEROLE holder may `COMMENT ON ROLE` every role it created, so it could hand the chart for deletion a role the chart never created. It also adopted any role that already existed under a declared name, so a role created under a name an administrator declares later stayed under its creator's control, for example `debezium` before `users.debezium.replication: true`, or `newdb_admin` before `databases.newdb`. Neither case can be caught from the catalog. Creating a role through an intermediate role and then dropping that role leaves the squatted role with no row in `pg_auth_members` at all, while the objects, role settings and default privileges planted in it stay. On 17.11 such a `debezium` had zero memberships, and after the chart adopted it `migrator` could still run code as `debezium` with `REPLICATION` through a function it had planted. The chart therefore keeps its own record: - Chart-managed roles are members of `cozystack:managed`, a NOLOGIN role that only the superuser can grant. On 16+ the membership is `WITH INHERIT FALSE, SET FALSE`. The name is outside the username and database charsets. The delete passes act only on these members. On its first run the chart marks every role it had commented before, when no role could yet hold CREATEROLE through it. - Once any user has held `createRole`, recorded as `cozystack.createrole_granted` on the `postgres` database (the same mechanism as `cozystack.adopted_restore`), the chart refuses to adopt an existing role it did not mark, both for users and for `<db>_admin`/`<db>_readonly`. The init job fails with the role's name, and a platform administrator must drop the role. Before that point adoption works as before, so a fresh install that declares `users.app` still adopts the role CNPG bootstraps. ## The PostgreSQL 16 floor on `createRole` The floor is a security boundary, not a cosmetic check. Before PostgreSQL 16, `CREATEROLE` allowed granting membership in any non-superuser role, so a user holding it could grant itself `pg_execute_server_program` and reach command execution on the instance, which is exactly what not exposing the superuser is meant to prevent. I verified both sides on a real cluster: on 15.15 `GRANT pg_execute_server_program TO <createrole_user>` succeeds and `COPY ... FROM PROGRAM 'id'` then returns `uid=26(postgres)`; on 16.11 and 18.1 the same grant is refused because it now requires ADMIN option on the role. PostgreSQL 16 restricted `CREATEROLE` to roles the grantee administers, which is what makes the attribute safe to expose. The guard uses the same fail-fast `SECURITY:` idiom as the blocked `postgresql.parameters` list in `templates/db.yaml`. Note that a grant obtained on PostgreSQL 15 survives a major upgrade: after moving an instance to 16.11 the user kept its membership in `pg_execute_server_program` even though new attempts were refused. Operators who exposed `CREATEROLE` out of band on an older major should audit and revoke memberships in `pg_execute_server_program`, `pg_read_server_files` and `pg_write_server_files` before relying on the floor. ## Verification The marker, adoption and `createRole` paths were rendered and run against live PostgreSQL 13, 15, 16 and 17 containers. An upgrade from the previous chart marks the roles it already managed and removes a dropped user. What happens to a dropped role's objects comes from #4574 and is covered by its tests. A fresh install adopts the CNPG `app` role in the same run that grants `createRole`. A forged comment is ignored, and squatted `debezium` and `newdb_admin` roles stop the init job with nothing granted to them. The orphaned-role pass still drops the roles of a removed database, and every re-run is clean. Reverting each part of the fix in turn fails `tests/managed_roles_test.yaml`. Beyond the chart unit tests, I replayed a real migration checklist against a live PostgreSQL 17.7 instance, connected as the managed login user and never as the superuser: creation of four roles with their attributes including `CREATEROLE` and `BYPASSRLS`, `CREATE SCHEMA ... AUTHORIZATION` landing on the requested owner, trusted extensions into a named schema, per-role and per-database `search_path`, per-role `statement_timeout` and `idle_in_transaction_session_timeout`, RLS including `FORCE ROW LEVEL SECURITY` verified end to end (a plain role sees zero rows, the `BYPASSRLS` role sees them all), and `SECURITY DEFINER` functions. All of it succeeds without the superuser. The one thing that remains out of reach is `CREATE EVENT TRIGGER`, which PostgreSQL restricts to superusers and no chart flag can change. Other attributes and the predefined roles that a managed-service admin role usually wants (`pg_monitor`, `pg_read_all_settings`, `pg_maintain`) can follow the same per-user pattern in a later PR, minus the three that grant OS access (`pg_execute_server_program`, `pg_read_server_files`, `pg_write_server_files`). ### Screenshots No UI change. ### Downstream repositories `packages/apps/postgres/values.schema.json` gains two fields, which per the trigger map reaches `cozystack/terraform-provider-cozystack`: its postgres resource carries a hand-maintained schema plus an expand/flatten pair for `users`, with no codegen from `values.schema.json`, so `createRole` and `bypassRls` will be invisible to Terraform users until they are added there by hand. I searched that repository and found no open PR or issue covering it (the only open PR is a dependabot Go module bump). I have not filed the follow-up, so I am leaving the box unticked rather than claiming work that does not exist. Happy to open the issue or the provider PR if a maintainer confirms that is the preferred route. I walked the rest of the map against the diff and found nothing else: no package is added, renamed or removed, no version enum changes, no default changes, `release.prefix` and the output Secret and Service names are untouched, and no CRD outside the regenerated `packages/system/postgres-rd/cozyrds/postgres.yaml` is affected. The `README.md` parameter table is regenerated here and postgres is already in the website generator's app list, so the website path stays automated. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: not filed yet, see the note above - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(postgres): add `users.<name>.createRole` and `users.<name>.bypassRls`, which grant the CREATEROLE and BYPASSRLS attributes to a managed user so a tenant can create and manage its own roles, including roles that bypass row-level security, without the postgres superuser. `createRole` also sets `createrole_self_grant` on the user so that roles it creates can own objects. `createRole` requires PostgreSQL 16 or newer: on older majors CREATEROLE can be used to grant oneself pg_execute_server_program and reach command execution on the instance, so the chart refuses to render the attribute there. The init job now records the roles it manages as members of a superuser-only `cozystack:managed` role instead of trusting role comments. Once `createRole` has been used on an instance, it refuses to adopt an existing role it did not create. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * PostgreSQL user configuration now supports `createRole` and `bypassRls` privileges. * These privileges can be combined with replication access. Enabling `createRole` also configures the role’s self-grant behavior. * `createRole` requires PostgreSQL 16 or newer; configuring it on older versions is rejected. * **Documentation** * Updated configuration schemas, examples, and guides to describe the new options and PostgreSQL version requirements. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
f5325425d6 |
fix(vm-instance): supply cloud-init network-data to netplan guests (#4306)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->
## What this PR does
A guest whose interfaces are rendered by netplan ignores two things the
platform already puts on the wire for it, and this adds an opt-in
`netplanDhcp` value that supplies the cloud-init network-data making it
ask for both.
`virt-launcher` starts a DHCP server on **every** bridge-bound
interface, secondary ones included — `newDHCPConfigurator` in
`pkg/network/setup/launcher/configurator.go` gates only on `iface.Bridge
!= nil`, and `BridgeConfigGenerator.Generate` serves the address the CNI
put on the pod-side interface along with that link's MTU. The chart
renders `bridge: {}` for every network, so a VM joined to a VPC subnet
is offered its reserved address and its MTU. cloud-init's fallback
network configuration covers a single interface, so on Debian and Ubuntu
nothing ever sends a DHCPDISCOVER on the second NIC and the interface
stays inert, while AlmaLinux (NetworkManager auto-profile) and Windows
come up. That asymmetry is the tell: with no DHCP server at all, none of
the three would work.
The same DHCP server advertises the launcher pod's search domains as
option 119, and `UseDomains=` defaults to `no` in systemd-networkd, so a
netplan-managed link keeps none of them. A cluster domain under `.local`
then reaches the mDNS exclusion in resolved and every lookup answers
`REFUSED`, which is #2085. `dhcp4-overrides: use-domains: true` is
enough on its own: the domains on the wire already carry whatever
cluster domain the cluster actually uses, so nothing needs to be
hardcoded here and #2372 does not collide with this.
### This is a regression, not a missing feature
`packages/apps/virtual-machine` carried exactly this, added in
|
||
|
|
bcda5c893a |
feat(kubevirt)!: drop HotplugVolumes for declarative volume hotplug (#4453)
## What this PR does This turns off the deprecated KubeVirt `HotplugVolumes` gate and moves the platform to declarative volume hotplug. It changes how every volume hotplug on the platform works, so the title carries `!`. Closes #4032. This PR is stacked on #4451 and has to merge after it. #4451 adds the kubevirt-csi flag this PR flips. The replacement gate is already on in KubeVirt 1.9, and the old gate is the only thing keeping it off. `DeclarativeHotplugVolumes` is Beta at v1.9.0, and Beta gates are on by default. virt-controller skips `handleDeclarativeVolumeHotplug` while `HotplugVolumes` is on, so dropping the old gate is what switches the new one on. The kubevirt-csi fallback goes off in the same change. With the old gate off, virt-api rejects the VMI-level `removevolume` on a VM-owned VMI, so the chart now passes `--enable-vmi-hotplug-fallback=false`. The comments next to the gate and the flag point at each other. This needs the rebuilt kubevirt-csi image. A driver built before #4451 does not know the flag and exits on it at startup. The image is stamped at release preparation and ships in the same release as this change, so only a `make apply` from main before then would crash-loop the CSI controller. What changes at runtime on an upgrade: - The VM-level `addvolume` and `removevolume` calls now edit `VM.spec` directly. virt-controller then makes the VMI hotplug volumes match it, instead of going through `VM.status.volumeRequests`. - Volumes hot-plugged through the VM subresource are in `VM.spec`, so they stay attached. - During a tenant teardown, when the VM goes before its VMI, the driver no longer calls the VMI subresource. It waits for the VMI status to drop the volume, and the garbage collector deletes the VMI with its VM. Upstream waits up to two minutes. If the VMI is still there after that, the unpublish fails and is retried. A new platform migration keeps old VMI-only volumes attached. On its first pass virt-controller removes every hotpluggable VMI volume that is missing from `VM.spec`. Until July 2025 kubevirt-csi attached volumes at the VMI level (the Role then granted `virtualmachineinstances/addvolume`), and live migration carries such a VMI across KubeVirt updates. Without help, a tenant node not recreated since then would lose a mounted disk on upgrade. Migration 59 copies each such volume and its disk from the VMI into the VM template before anything else changes. Adding a hotplug volume to the VM spec is a valid live update in KubeVirt, so no VM needs a restart. It runs as the platform chart's pre-upgrade hook, before the upgrade re-points `cozystack-packages`. The KubeVirt CR and the tenant kubernetes charts both render from that source, so the gate and the flag cannot change before the migration has finished. It only touches VMs that need it, logs each one, and a second run changes nothing. The migration also makes ephemeral hotplugs permanent. A volume a user added at the VMI level, for example with `virtctl addvolume` without `--persist` from an older virtctl, has the same shape as kubevirt-csi's old attachments, because virt-api builds both the same way. There is nothing reliable to tell them apart by, so both are copied. Leaving the user's volume out would not keep it ephemeral: the first reconcile would unplug it from the running VM. After the upgrade it is part of `VM.spec` and attaches again on a restart, and `virtctl removevolume` drops it. Ephemeral hotplug stops working on VMs. With the old gate off, virt-api refuses VMI-level `addvolume` and `removevolume` on a VMI that a VM owns (`vmiVolumePatch` in v1.9.0). A volume on such a VMI is added and removed through the VM from then on. A VMI without a VM is not affected. Only running VMIs are copied. virt-controller reconciles no other VMI, and a stopped VM builds its next VMI from `VM.spec`, so copying from a stopped one would bring back volumes the restart was going to drop. Migrations run only on the isp-full, isp-full-generic and isp-hosted variants. KubeVirt comes from the platform only on isp-full and isp-full-generic, so those are covered. On the default variant, where packages are installed by hand, nothing runs the migration. This optional check lists the VMIs and volumes the migration would copy. It uses the same rule. On the default variant, run it before upgrading and recreate each tenant node it lists: ```bash kubectl get vm,vmi --all-namespaces --output json | jq --raw-output ' ([.items[] | select(.kind == "VirtualMachine") | {key: "\(.metadata.namespace)/\(.metadata.name)", value: [.spec.template.spec.volumes[]?.name]}] | from_entries) as $vm | .items[] | select(.kind == "VirtualMachineInstance") | ([.metadata.ownerReferences[]? | select(.controller == true and .kind == "VirtualMachine" and (.apiVersion | startswith("kubevirt.io/"))) | .name][0]) as $owner | "\(.metadata.namespace)/\($owner)" as $key | select(.status.phase == "Running" and $owner != null and $vm[$key] != null) | .spec.volumes[]? | select(.dataVolume.hotpluggable == true or .persistentVolumeClaim.hotpluggable == true) | .name as $n | select($vm[$key] | index($n) | not) | "\($key) \($n)"' ``` Empty output means nothing is affected. I checked the query against the migration's test fixtures only. Neither the query nor the migration has run on a live cluster. #4287 also adds a migration 59. If it merges first, this one moves to 60: the file, the version it stamps, `migrations.targetVersion`, the bats file (its `run_migration` argument and expected `STAMP`) and `hack/testdata/migration-59-vmi-hotplug`. A VM with a pending hotplug request at upgrade time may also get stuck. virt-controller still processes `VM.status.volumeRequests` before the declarative handler, and it applies them through the VMI subresource, which virt-api rejects once the old gate is off. The sync then fails before the VM spec is updated, so nothing clears the request, and every later sync of that VM fails the same way. A retry through the VM subresource edits the spec and clears it, which covers kubevirt-csi's own requests. A one-shot request stays stuck until the VM restarts. I read this from the v1.9.0 code and haven't reproduced it. Waiting until this lists nothing before upgrading narrows the window, but a request made during the rollout can still hit it: ```bash kubectl get vm --all-namespaces --output json | jq --raw-output '.items[] | select((.status.volumeRequests // []) | length > 0) | .metadata.namespace + "/" + .metadata.name' ``` The two halves live in different HelmReleases, so there is a short window during the rollout where the driver and the gate disagree. In that window an unpublish fails and is retried until both are updated. A tenant cluster whose `kubernetes` HelmRelease is suspended or failing keeps its old driver for longer. Drivers from before July 2025 attach through the VMI subresource, so on such a tenant every attach is refused until its release reconciles. Such a driver can also attach a volume after the migration has run and before the gate flips. The migration doesn't see that volume, and the first declarative reconcile unplugs it from the running VM. The migration lists the VMs once and patches each one with a `resourceVersion` test, so a VM that changes in between fails the hook Job. The Job has `backoffLimit: 3`, and each retry lists again and skips the VMs already patched. Re-reading only the conflicting VM would avoid the retry; I left that out. The new gate is listed in the CR even though it is on by default. With the old gate gone it is the only hotplug path, and the chart refuses a render that disables it. I found no other user of `HotplugVolumes` in the platform or in the downstream repositories. The VMI-level fallback may have been failing already. The kcsi Role has no rule for the VMI-level `removevolume` (#4452), so that call likely got 403 with or without the gate. I haven't checked that on a cluster. `hack/migration-59-vmi-hotplug-volumes.bats` runs the migration against a fake kubectl: a VM that needs the copy, VMs and VMIs that must be left alone (a VMI that is not Running among them), a second run, a cluster without KubeVirt and the fail-closed paths. The helm-unittests for the KubeVirt CR pin the whole gate list, accept `HotplugVolumes` in `disabledFeatureGates` and refuse `DeclarativeHotplugVolumes` there. The CSI test pins the flag at false, and the `make update` idempotency test still passes. None of that exercises hotplug itself. The real test is e2e with volume attach and detach on a tenant cluster, including a teardown. No CI lane upgrades from the previous release yet, so an upgrade with a live VMI-only volume is untested; #4463 tracks that test. ### Screenshots No UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(kubevirt)!: KubeVirt now runs with DeclarativeHotplugVolumes instead of the deprecated HotplugVolumes feature gate, and kubevirt-csi no longer uses the VMI-level hot-unplug. Volume hotplug requests are applied through VM.spec. The upgrade copies volumes that older kubevirt-csi versions attached only to the VMI into VM.spec, so they stay attached; the check in the PR description is optional. This also applies to ephemeral volumes added to running user VMs at the VMI level, such as with virtctl addvolume without --persist: they become part of VM.spec and attach again after a VM restart. virt-api now refuses VMI-level addvolume and removevolume on a VMI owned by a VM, so such volumes are added and removed through the VM. A tenant cluster whose kubernetes release is suspended or failing keeps its old kubevirt-csi, and a driver from before July 2025 cannot attach volumes there until that release reconciles, and a volume it attaches during the upgrade itself can be unplugged again. A values override that puts DeclarativeHotplugVolumes into disabledFeatureGates now fails the kubevirt render. Clusters on the default variant do not run platform migrations and should run that check before upgrading. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Declarative hotplug volumes are enabled by default for KubeVirt. * Running virtual machines keep eligible hotplugged volumes and disks when their configuration is migrated. * **Bug Fixes** * The migration avoids adding duplicate volumes or disks and does not mark the migration complete if required operations fail. * The Kubernetes CSI driver no longer uses the VMI hotplug fallback. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
ff3c409881 |
fix(backups): refuse a foreign -orig PVC and record the kept disk's reclaim policy (#4671)
## What this PR does Two defects in `renamePVC` (`internal/backupcontroller/velerostrategy_controller.go`), the step of an in-place VMInstance restore that keeps each disk as `<dv>-orig-<shortHash(RestoreJob name)>` when `keepOriginalPVC` is true, the default. ### A taken `-orig` name skipped the rename, and the restore went on `renamePVC` returned `nil` as soon as a PVC with the `-orig` name existed (`main:1263-1266`), whatever that PVC was. `prepareForRestore` then reached Step 4 (`main:1172-1183`), which deletes the DataVolume of every disk regardless of what Step 3 did, because a Step 3 error was only ever turned into a Warning event (`main:1165-1168`). The restore therefore ran exactly as `keepOriginalPVC: false` does and reported `Succeeded`: the live disk was not kept, and the `-orig` name held a different, older disk that the user takes for the pre-restore copy. When that name can already exist: - A RestoreJob created under the name of a deleted one. The hash covers only the RestoreJob name, and nothing in the controller ever deletes an `-orig` PVC, so the predecessor's PVC is still there. This is the realistic case, and it is what a scripted restore with a fixed RestoreJob name runs into on its second run unless the `-orig` PVC was removed in between. - Two RestoreJobs with different names whose 4-hex-digit hashes collide (16 bits) while restoring the same disk in the same namespace. Rare, but nothing guards it. - A reconcile of the same RestoreJob after a controller restart in the middle of a rename. The name is ours, but the early return treated a half-done rename as complete: the original PVC stayed in place and the PV stayed bound to it. - Truncation does not apply: the name is not truncated, and one longer than 253 characters makes the `Create` fail, an error that was swallowed in the same way. Each `-orig` PVC now carries the UID of the RestoreJob that created it, in `backups.cozystack.io/restore-job-uid`. `prepareForRestore` checks every `-orig` name before it suspends or halts anything, and fails the RestoreJob, naming the PVC, when any of them is held by a PVC another RestoreJob created. When the PVC is ours, `renamePVC` resumes: it deletes the original PVC only while it still claims the kept PV, and repoints the PV unless it already points at the `-orig` PVC, so a rerun is a no-op. Any other rename error now holds the restore back before Step 4 deletes a DataVolume, and the restore is retried, instead of the error becoming an event (see the next section). An `-orig` PVC created by an earlier controller has no UID annotation and is refused like any foreign PVC. The only RestoreJob that loses by this is one whose controller was upgraded after its rename and before its Velero Restore was created; it fails with the message above rather than going on. ### The rename read its own write back through the cache `renamePVC` created the `-orig` PVC, then read it back through the cached client to learn its UID for the PV's claimRef (`main:1314-1316`). The informer often has not seen a PVC created a moment earlier. On our lab, running Cozystack v1.6.3, an in-place restore of a two-disk VM with `keepOriginalPVC: true` renamed and restored the system disk, while the data disk lost the race, with this event on the RestoreJob: `Failed to keep old PVC vm-disk-e2e-dc-fix-data: failed to get -orig PVC vm-disk-e2e-dc-fix-data-orig-a574: PersistentVolumeClaim "vm-disk-e2e-dc-fix-data-orig-a574" not found`. As the error was only an event, the data disk's DataVolume was deleted and its original PVC stayed in place. Velero skipped that PVC and created no DataDownload for it, so its content was not restored, yet the restore finished `Succeeded`. The `-orig` PVC was left `Pending` on the same PV, which was by then `Retain`. `Create` already fills in the UID, so the read-back is gone. The other reads the rename decides on (the `-orig` PVC, the original PVC and the PV, before and after its own writes) go through the manager's uncached reader, wired as `APIReader`. The PV rebind retries on conflict, since the PV controller also writes the PV once the original PVC is deleted. A rename that fails for any reason other than a foreign `-orig` name requeues the restore rather than failing it. The DataVolume is still there, and the retry finds its own `-orig` PVC by the RestoreJob UID and resumes where it stopped. A cause that never clears keeps the RestoreJob `Running`, with a Warning event on each attempt. #4673 adds the same `APIReader` field and `apiReader()` helper to `RestoreJobReconciler`, for its finished-job guard. Whichever of the two lands second drops its own copy when it rebases. ### The PV's reclaim policy was overwritten without being recorded `renamePVC` switches the PV to `Retain` so it survives the deletion of the original PVC (`main:1289-1293`), and the policy it had was lost. I looked for a path in the controller that releases or cleans up `-orig` PVCs, and there is none: `cleanupOnDelete` on a Velero RestoreJob removes only the Velero `Restore` and the resource-modifier ConfigMaps, and no code deletes an `-orig` PVC or touches its PV again. A PV that was `Delete` therefore stays `Retain` for good, and its volume leaks when an operator deletes the `-orig` PVC. The original policy is now recorded in `backups.cozystack.io/original-reclaim-policy`, on the PV and on the `-orig` PVC, following the `backups.cozystack.io/` prefix the controller already uses for `underlying-resources`. A value already present on the PV is kept, as it predates any rename. Since there is no release path to restore it from, `examples/backups/vmi/scenario-user-restore.md` gains a section for the operator who deletes an `-orig` PVC: set the recorded policy back on the PV first, with a snippet that leaves a PV without the annotation on `Retain`. Restoring the policy from the controller itself, for instance once the Velero Restore has completed and the PV is `Bound` to the `-orig` PVC, would close the leak without the operator. I did not do it, because it changes what deleting an `-orig` PVC destroys, from nothing to the volume; that is a maintainer call, and the annotation is what it would read. ### How this fits with #3427 #3427 is complementary, and the two do not touch the same code: it changes `collectUnderlyingResources`, this PR changes `prepareForRestore` and `renamePVC`. Both append tests to `velerostrategy_controller_test.go`, so whichever lands second may need a trivial rebase. - #3427 makes a standalone VMDisk report its own disk, which sends VMDisk in-place restores through this same `renamePVC`. Once both land, the VMDisk path gets the same guard and the same annotation. - Its `examples/backups/{vmi,vmdisk}/cleanup.sh` and the Chainsaw `finally` reclaim each `-orig` PVC by patching its PV to `Delete` before deleting the PVC. That works around, in the e2e teardown, the leak described above. It is correct on the LINSTOR classes, which are `Delete`, but it would destroy a volume whose class was `Retain`. With this PR the scripts can read `backups.cozystack.io/original-reclaim-policy` instead of hard-coding `Delete`. I have not changed them here, since they only exist on that branch. - Its round-trips use fixed RestoreJob names and assert that a `<pvc>-orig-*` PVC exists after the restore. If a failed teardown leaves one behind, today the next run skips the rename and the assertion still passes on the stale PVC; with this PR the RestoreJob fails and names it. - Its cleanup comments describe the `-orig` PVC as a bare ObjectMeta with no labels or owner. That stays true: this PR adds annotations only, so its name-based lookup is unaffected. ### Tests New tests in `internal/backupcontroller/velerostrategy_controller_test.go`, run through `prepareForRestore` with the DataVolume in the dynamic fake client: - `TestPrepareForRestore_KeepOriginalPVC_RefusesForeignOrigPVC`: an `-orig` PVC without the annotation, and one carrying another RestoreJob's UID, both fail the restore with the PVC named, and leave the DataVolume, the live PVC, the PV and the foreign PVC untouched. - `TestPrepareForRestore_KeepOriginalPVC_ResumesOwnRename`: a rename interrupted after the `-orig` PVC was created completes on the next pass, and a second pass changes nothing. - `TestPrepareForRestore_KeepOriginalPVC_FailedRenameKeepsDataVolume`: a rename that fails for another reason (missing PV) requeues the restore without failing it and keeps the DataVolume, and once the PV exists the retry completes the rename. - `TestPrepareForRestore_KeepOriginalPVC_CacheLagsBehindWrites`: the reconciler's cached client serves every read from the objects as they were seeded, through an `interceptor` over the fake client, as an informer that has not caught up with this reconcile's writes would, while `APIReader` sees them. The rename must complete. - `TestRenamePVC_RecordsOriginalReclaimPolicy`: a `Delete` PV and a `Retain` PV both end up `Retain`, with their original policy recorded on the PV and on the `-orig` PVC. With the controller reverted to `main` (only the two annotation constants kept so the tests compile), all four fail: `prepareForRestore() = ready true, nil error` with `DataVolume was deleted although the disk could not be kept` for both foreign cases and for the failed rename, `original PVC still exists` and the PV still bound to it for the interrupted rename, and an empty `original-reclaim-policy` on the PV and on the PVC. For the cache race, the first commit alone fails `CacheLagsBehindWrites` with the lab's error, `failed to keep PVC vm-disk-ubuntu-source: failed to get -orig PVC vm-disk-ubuntu-source-orig-de3e: persistentvolumeclaims "vm-disk-ubuntu-source-orig-de3e" not found`, and `FailedRenameKeepsDataVolume` with a terminal error instead of a requeue. On `main` the race test reproduces the lab outcome: `original PVC still exists`, and the PV still claimed by it. With both commits, `go test -count=1 ./internal/backupcontroller/...` passes. No API type, CRD, chart or values file changes, so there is nothing to regenerate. ### Screenshots N/A, no UI change. ### Downstream repositories Walked the trigger map against the diff: the restore preparation in `internal/backupcontroller/velerostrategy_controller.go`, its tests, and `examples/backups/vmi/scenario-user-restore.md`. No CRD, ApplicationDefinition, values, schema, enum, namespace, label, metric or tooling change, so terraform-provider-cozystack, ansible-cozystack, ccp, talm, cozyhr, cozy-proxy, cozystack-telemetry-server, external-apps-example, examples and community are not reached. `RestoreJob.spec.options` stays opaque to the provider. cozystack/website is reached: `content/en/docs/next/virtualization/backup-and-recovery.md` (and each released copy) tells users to delete the `*-orig-*` PVCs "to free up storage". A PV left on `Retain` frees nothing, and the page should explain the recorded policy and the new failure for a reused RestoreJob name. No follow-up exists there yet, so the box stays empty until one is filed and linked. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): an in-place VM restore with keepOriginalPVC now fails, before the VM is halted, when the <disk>-orig-<hash> PVC it would keep the disk under already exists and was not created by this RestoreJob (for instance one left by a deleted RestoreJob of the same name), instead of skipping the rename and restoring without keeping the current disk. Any other failure to keep a disk, including a race in which the rename read its just-created -orig PVC back through the cache and failed, now holds the restore back and retries it before its DataVolume is deleted. Previously such a restore reported success without restoring the disk. The PV's reclaim policy before the switch to Retain is recorded in the backups.cozystack.io/original-reclaim-policy annotation on the PV and the -orig PVC; set it back on the PV before deleting the -orig PVC, or its volume is not reclaimed. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * In-place restores that preserve original PVCs can now resume safely after interruptions, including when reads lag behind recent changes. * Restores stop before suspending the VM if a conflicting `-orig` PVC already exists. PVC rename failures can be retried, while conflicting names remain fatal. * Original PV reclaim policies are recorded and restored before preserved PVCs are deleted. * **Documentation** * Updated the restore guide with disk-reclamation steps and guidance for resolving leftover PVC name conflicts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |