Commit Graph
522 Commits
Author SHA1 Message Date
Myasnikov Daniil 30ee57f892 fix(operator): route every downward ceiling move through the grace period
The grace period guarded only the branch where the scan came back empty,
which left the same defect standing in the branch where it finds
something smaller than the ceiling already in place.

A namespace raised to 16Gi for a pod visible only as a live Pod, with a
second oversized pod at 8Gi beside it, loses the larger pod to a node
drain. The scan still finds the 8Gi one, so the empty-scan guard never
engages, and the ceiling was written straight down to 8Gi -- rejecting
the larger pod's recreation exactly as writing the configured limit
would have. A blocker smaller than the standing ceiling is not evidence
that a larger one is gone.

So the rule is now about direction rather than about why: the ceiling
the evidence justifies is computed first, and any move that lowers it
goes through the grace period, whichever of the three reasons prompted
it -- an empty scan, a smaller blocker, or the configured limit itself
being lowered beneath a standing ceiling. The eventual drop lands on
what the scan justifies rather than on the configured limit, so a
smaller blocker that is genuinely still there keeps the ceiling it
needs.

The raise is also no longer logged when it changes nothing. A namespace
holding a permanently oversized pod re-justifies the same ceiling on
every reconcile, and announcing that put a line in the log for every
such namespace on every HelmRelease status change.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:55 +03:00
Myasnikov Daniil 9fe93a3d7e docs(operator): fix three claims that overstated what the code does
The static-pod note said containers built from an on-disk manifest are
unaffected "in both directions". True of the container, which the
kubelet runs from disk whatever the API says, but the mirror pod is an
ordinary API object that passes admission like any other, so a static
pod declaring a memory request above the ceiling with no limit of its
own has its mirror pod rejected -- leaving the container running with no
representation in the API. Worth naming, since kube-system is in scope
by design. The scan does cover it: a mirror pod is a live Pod.

The ReplicaSet exclusion was justified on live ones being copies of
their Deployment's template. That does not reach an ownerless
ReplicaSet, which at replicas 0 has neither pods to be seen through nor
a Deployment template standing in for it. It stays out of scope --
nothing here ships a bare ReplicaSet -- but as a gap left open rather
than one the stated reason closes.

The delete guard's comment claimed the enabled apply always adopts and
stamps a same-named foreign object. It does not when that object's spec
already matches, because then the apply is skipped entirely. The
outcome is the same one the guard gives on purpose, but it is reached by
the apply not running.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:54 +03:00
Myasnikov Daniil e131a49514 feat(operator): default the system namespace memory ceiling to 32Gi
The guard in front of this default reads declared memory requests. It
cannot see a container's real working set, so a system component that
requests little but legitimately uses more than the ceiling, and
declares no limit of its own, was handed a deterministic kernel OOM on
its next recreation in exchange for leaving the Talos handler's victim
set. Since the feature ships enabled, that trade fell on the whole
fleet on upgrade.

It is not a trade this feature has to make. The handler discards a
cgroup for carrying a memory.max at all, whatever its value, so
immunity does not require a fitted ceiling -- only a real one. A
default far above any plausible working set buys the immunity and
leaves nothing for the ceiling to bind on, which also keeps it clear of
the failure mode this repository already carries a scar from, where an
undersized limit converted a rare pressure-driven kill into a reliable
one.

Nothing about scheduling changes: only requests are scheduled against,
and the paired default request stays 32Mi. An operator who wants the
ceiling to bind lowers it, per component in a chart or fleet-wide with
this knob, once real usage is measured -- which is the direction that
needs evidence, and now the direction that needs it rather than the
default.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:53 +03:00
Myasnikov Daniil ad3c9ef04e fix(operator): stay out of a namespace another LimitRange governs
The reconcile only read its own LimitRange by name, so a system
namespace already carrying an administrator's LimitRange got a second
one beside it. LimitRanger applies defaults from every LimitRange in
the namespace and then validates the pod against every one, and nothing
orders them, so the effective container ceiling moved between the
administrator's value and this one for reasons no chart author could
see. That is the same hazard this feature cites to keep itself out of
tenant namespaces, and it does not stop being one because the namespace
is a system namespace.

A max is the sharper half. This reconciler writes a default and no max,
so an administrator's LimitRange carrying a container memory max below
the configured default had every container that declares no limit of
its own rejected: the default is written first, then validated against
the max it exceeds. That takes out a whole namespace, and it reaches
kube-system, which this feature enters for cozystack-scheduler and
which is where a distribution's or an administrator's own LimitRange is
likeliest to sit.

Such a namespace is now left alone and logged, and anything this
reconciler wrote there earlier is withdrawn rather than left sitting
beside the foreign object, since that object is half of the ambiguity.
The check is deliberately narrow -- only a container memory default or
max collides with what is written here -- because read more widely it
would be a blanket opt-out that an unrelated cpu policy could trip.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:52 +03:00
Myasnikov Daniil 5630954e34 fix(operator): hold a raised memory ceiling before lowering it
A raised container memory ceiling was recomputed from the scan on every
reconcile, so a single scan that found nothing lowered it back to the
configured limit. An empty scan is not evidence that the oversized
request is gone: it is equally consistent with the pod being between
deletion and recreation.

For a blocker that appears only as a live Pod, which is a class this
repo ships -- every CNPG Cluster in a system namespace builds its
instance pods directly rather than through a StatefulSet -- an ordinary
Talos node drain opens that window, and reconciles follow HelmRelease
status churn, of which a node upgrade produces plenty. The ceiling was
lowered under a pod that was merely being rescheduled; its recreation
was then defaulted a limit below its own request and rejected at
admission for good, because the rejected pod never exists so no later
scan can find it, and the namespace settles at the configured limit
where the pod half of the scan stops running at all. The drop was not
even logged.

Lowering now waits until the same emptiness has held for
memoryCeilingDropGrace, with the instant it was first observed recorded
on the object so it survives an operator restart and a change of
leader. A wall clock rather than a count of consecutive clean scans,
because the reconcile rate rises exactly when the risk does and any
small count can be burned inside one rescheduling window. A blocker
seen again cancels the countdown, an unreadable stamp restarts it, and
both the hold and the drop are logged.

The steady-state comparison now also covers the two metadata fields
this reconciler manages, because the grace stamp is one of them. That
incidentally fixes the managed-by label never being re-stamped when
only metadata had drifted, which had left turning the knob off unable
to remove an object the operator did own.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:52 +03:00
Myasnikov Daniil 3c1d8d6988 docs(operator): bound the managed-by guard to the disabled state
The comment on the label check in deleteSystemDefaultsLimitRange reads as
a general promise not to touch an administrator's own LimitRange of that
name. It is narrower than that, and the difference is worth writing down:
while the feature is enabled the apply takes the name over with
ForceOwnership and stamps the managed-by label itself, so a hand-written
cozystack-system-defaults is adopted and then deleted here when the knob
goes off.

That is the intended SSA behaviour, not a defect. What the label actually
buys is that a LimitRange of that name the reconciler never wrote to
survives the knob, which is a real guarantee and a smaller one.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:51 +03:00
Myasnikov Daniil accb4f9e1b refactor(operator): drop the dead error check from the settled-state guard
The guard read `err == nil && apiequality.Semantic.DeepEqual(...)`, which
looks like it consults the earlier Get and does not. That Get's error was
reassigned by the scan two statements up, and the scan's non-nil case has
already returned, so the clause is always true by the time it is
evaluated.

It was harmless only by accident: a namespace whose Get returned NotFound
carries an empty spec, which never compares equal to the desired one, so
the apply happened anyway. Removing the clause makes that the reason
rather than a coincidence sitting behind a condition that cannot fire.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:50 +03:00
Myasnikov Daniil ead330ccc3 fix(operator): keep scanning pods while the ceiling stays raised
The pod half of the memory scan was skipped whenever any container memory
default was in force. That is the weaker of the two available conditions,
and it silently retracted a ceiling this reconciler had just raised.

The default written into a system namespace is either the configured limit
or a ceiling raised above it to clear an oversized request, and the raise
is recomputed from the scan every reconcile rather than read back off the
object. So a raise that only the pod half could justify -- a live pod
requesting more than the limit with no limit of its own, in no workload
template the scan can read, which is what an operator building pods from
its own custom resource produces -- lasted exactly one reconcile. The
raised LimitRange satisfied "in force", the next reconcile skipped the pod
list, the scan came back empty, and the configured limit was reapplied
underneath a still-running pod. That pod's next recreation was rejected
with "must be less than or equal to memory limit", and because the skip
stayed in effect nothing ever raised the ceiling again: a healthy
component goes down at a node reboot and stays down, with the revert
unlogged.

Lowering the knob reached the same place with no race in it. An operator
running at 16Gi has pods admitted with requests only that ceiling allowed;
dropping the flag to 4Gi wrote 4Gi straight over them.

Gate the skip on the namespace already defaulting memory at exactly the
configured limit instead. That keeps the pod list alive for as long as a
namespace sits raised, which is what lets the raise be re-justified or
dropped on evidence rather than frozen or forgotten, and it still skips
the list in the steady state the skip was added for. The comparison is by
value for the same reason the spec comparison is semantic: 4096Mi and 4Gi
are one ceiling.

The cost is that a raised namespace reads the pod list every reconcile.
That is the price of being able to lower the ceiling again, bounded by how
many namespaces sit raised -- none, once the oversized requests are gone.

Three tests cover it, each of which fails on the old condition: a raised
ceiling held while the pod needing it runs, the same ceiling dropping back
once that pod is gone, and the configured limit falling below a ceiling
already in place. A fourth pins the notation equality. The fixture comment
on the steady-state test claimed its 8Gi pod could not exist beside a
settled 4Gi LimitRange; it can, through a narrow window, so it now says
so.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:50 +03:00
Myasnikov Daniil 26a1ce5be0 test(operator): drop the incidental fixture from the no-pods case
The scaled-to-zero test seeded a spec-less LimitRange, which is not part
of the case it names. It is the first application in the namespace that
makes a workload with no pods invisible to a pod-only scan, so start from
no LimitRange at all and let the assertion rest on the workload rather
than on the fixture.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:50 +03:00
Myasnikov Daniil b62fac474c refactor(operator): move both server-side applies off the deprecated client.Apply
client.Apply as a patch type is deprecated in controller-runtime v0.23.1
in favour of Client.Apply taking a runtime.ApplyConfiguration. Convert
both call sites — the system defaults LimitRange, new on this branch, and
the Namespace apply that predates it — keeping the same field owner and
force-ownership options.

The LimitRange keeps its typed builder, because that is what the spec
comparison and the tests read, and gains a small function that restates
it as an apply configuration immediately before the wire. The Namespace
apply now sends only the labels and annotations the reconciler actually
sets, rather than a whole typed object whose spec and status were empty.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:49 +03:00
Myasnikov Daniil c7af79b734 fix(operator): only delete the LimitRange this reconciler owns
deleteSystemDefaultsLimitRange matched on name alone, and the name is not
reserved: nothing stops an administrator creating a LimitRange called
cozystack-system-defaults by hand. Turning the knob off would then
silently remove policy this operator never wrote, in every system
namespace at once.

Stamp app.kubernetes.io/managed-by on the object the reconciler applies
and require it before deleting. Anything else is left where it is, which
is also the right answer if the name is ever taken over by another
component. Nothing in the wild carries the old unlabelled shape, since
the feature has not shipped.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:49 +03:00
Myasnikov Daniil dbc5e5ea13 perf(operator): read before deleting the disabled LimitRange
With the limit set to 0 the disabled path runs for every system namespace
on every Package reconcile and issued a DELETE each time, swallowing the
404 that nearly all of them returned. That is a write attempt per
namespace per reconcile, forever, and every one of them lands in the
audit log.

Get from the informer cache first and return when there is nothing there.
The test now counts deletes, so a second reconcile over an already-empty
namespace has to stay at zero rather than merely not erroring.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:48 +03:00
Myasnikov Daniil 89b78cbf20 docs(operator): drop the duplicated field from the ceiling-raise log
request and raisedLimit were logged with the same value, because the
raised ceiling is the blocking request. Keep request, which names the
thing an operator has to go and change, and say in the message that the
ceiling becomes it.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:48 +03:00
Myasnikov Daniil 87f79c17c3 fix(operator): keep scanning templates after the LimitRange settles
The steady-state fast path returned as soon as the LimitRange matched the
configured spec, on the grounds that LimitRanger gates anything a scan
could still find. That holds for pods and not for templates: a Deployment
added to a settled namespace with a memory request above the ceiling and
no limit of its own is admitted exactly as written, and only the pods it
then tries to create are rejected with "must be less than or equal to
memory limit". Nothing raised the ceiling, because every later reconcile
returned at the same line, so a chart bump became a rollout that could
never produce a pod and logged nothing about it.

Split the scan instead. Templates are read on every reconcile; the pod
list, which is the expensive half, is read only while no memory default
is in force in the namespace — once one is, LimitRanger fills in a limit
for every container admitted without one, so a container carrying a
request with no limit beside it can no longer come into existence. The
skip keys on the default actually being in force rather than on the
object existing, so a LimitRange whose spec was stripped scans pods again.

The write is now guarded by a semantic comparison against the spec the
scan settled on rather than against the configured spec, so a namespace
already sitting at a raised ceiling stops taking an identical apply on
every reconcile as well.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:46 +03:00
Myasnikov Daniil c9cacda1a0 docs(operator): pin the admission order by rule, not plugin index
The scan's justification for treating a VerticalPodAutoscaler as harmless
rested on LimitRanger being "position 6 of AllOrderedPlugins" and
MutatingAdmissionWebhook being "35". Both numbers are properties of one
release rather than of the ordering, and they had already drifted out
from under the comment. Under k8s 1.35, which this tree builds against,
LimitRanger is the 6th entry but MutatingAdmissionWebhook is the 33rd;
index 35 is ResourceQuota and the 35th entry is ValidatingAdmissionWebhook,
neither of which mutates, so the sentence named the wrong plugin under
either counting convention. In 1.31 the same plugin sat at 30, and
SecurityContextDeny being dropped in 1.35 is what moved LimitRanger.

Restate it on the invariant the list asserts about itself -- webhook,
resourcequota and deny plugins must go at the end -- with LimitRanger
compiled in above that marker and every plugin that can rewrite a
container's resources below it. That survives a release bump where an
index does not, and it covers MutatingAdmissionPolicy as well, a second
mutator that the positional wording predated and never accounted for.

Record what the order buys, which the old wording left implied. Run the
other way round, a plugin mutating before LimitRanger would write a
request into a container that has no limit yet, LimitRanger would then
default the limit to the configured ceiling, and core validation would
reject the fully mutated pod. The scan can see none of that -- the
request is in no template and the pod is never admitted -- so the order
is the only thing keeping VPA out of the residual gap this comment
already records for controlledValues: RequestsOnly.

Fix the matching rationale on the live-pod case in
TestFindRequestAboveDefaultLimit, which still claimed the pod's request
was raised by a webhook after the template was rendered. The ordering
rules that out; what the case actually rests on is a pod admitted before
the LimitRange existed, or one its controller builds from a custom
resource no template in the scan can show.

No behaviour change, and no new test: a container that carries its own
memory limit is already pinned out of the scan's scope for a template,
in TestFindRequestAboveDefaultLimit, and for a live pod, in
TestReconcileSystemDefaultsLimitRangeAppliesWhenLargeRequestCarriesItsOwnLimit.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:46 +03:00
Myasnikov Daniil 16b8ea4f0e docs(operator): correct the VPA rationale on the request scan
The comment argued that the danger this scan guards against is a VPA
webhook writing a request above the defaulted limit, which is then
rejected at admission. Admission ordering rules that out. LimitRanger is
position 6 of AllOrderedPlugins and MutatingAdmissionWebhook is 35, so
the LimitRange has already defaulted the container's memory limit before
VPA is called, and VPA rescales that limit in proportion to the request
it writes (GetProportionalLimit, under the default controlledValues:
RequestsAndLimits). Request and limit stay consistent and the pod is
admitted.

Rewrite it around what can actually happen: a request already in the spec
when it reaches admission with no limit beside it. In this tree that means
an operator lowering --system-namespace-memory-limit below a statically
declared request -- the largest is 2Gi, in
packages/system/rabbitmq-operator, so the 4Gi default clears them all
today. Two narrower shapes are real too: a pod that predates the
LimitRange, whose own controller may build it from a custom resource no
workload template in this scan can show; and a VPA set to
controlledValues: RequestsOnly, where nothing rescales the defaulted limit
and the rejection returns. The second doubles as the residual gap, since
a first pod rejected outright appears in neither a template nor a pod
list.

The scan itself is unchanged; only the justification was wrong.

Also record why withholding the LimitRange had to go rather than merely
being suboptimal: in a namespace with no LimitRange nothing defaults a
memory limit, so the next VPA-sized pod arrives with a large request and
no limit, the scan finds it again, and the namespace is withheld again.
The old behaviour could never get its foot in the door in cozy-monitoring.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:45 +03:00
Myasnikov Daniil d682158d1f test(operator): pin the tenant namespace boundary
One strings.HasPrefix keeps this feature out of tenant namespaces, and
nothing tested it: "tenant-" appeared twice in package_reconciler.go and
not once in the test file, so replacing the check with isSystem := true
left the whole package green.

What it protects is not subtle. packages/apps/tenant ships
tenant-range-limits, which defaults container memory to 128Mi in every
tenant namespace that declares resourceQuotas. A system LimitRange
landing beside it leaves two LimitRanges defaulting the same containers,
and which of them wins is decided by the order LimitRanger happens to
iterate them -- neither chart controls that -- so a tenant container's
default memory limit would move between 128Mi and 4Gi with nothing in the
tree changing.

Reconcile a variant with one system and one tenant namespace, and assert
no LimitRange is written for the tenant one. The system namespace in the
same run is the positive control: without it the test would still pass if
LimitRanges stopped being written anywhere at all. The cozystack.io/system
label is asserted too, since the same boundary gates both.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:44 +03:00
Myasnikov Daniil 2e1ab6682c fix(operator): raise the namespace ceiling instead of retracting it
When a container in a system namespace requests more memory than the
configured default, the guard used to withhold the LimitRange and delete
any it had applied earlier. That leaves the namespace with no default
memory limit at all, which is the precise state this feature exists to
remove: without memory.max on the pod cgroup the Talos OOM handler keeps
the pod in its victim set, and a tenant workload's memory pressure gets
paid for by a system component.

Default that namespace's memory to the offending request instead. A
looser ceiling still sets memory.max, so the pod scores zero in victim
selection exactly as it would at the configured limit -- 8Gi protects it
as well as 4Gi does, and both protect it infinitely better than nothing.
The raise needs no knowledge of where the request came from, and it
undoes itself: once the request is gone the scan finds nothing, desired
falls back to the configured limit, and the next reconcile applies it.

This is not a hypothetical namespace. packages/system/monitoring pins
maxAllowed.memory at 8Gi for vmselect and vmstorage,
packages/system/monitoring-agents pins 6G for vmagent, all three run
updateMode: Initial, and no monitoring template declares a memory limit.
Initial never evicts, so those requests do not come down on their own:
under the old behaviour cozy-monitoring, which holds the largest memory
consumers on the platform, was permanently excluded from the feature on
exactly the clusters it was written for.

A raised namespace no longer matches the base desired spec, so it scans
on every reconcile. That is deliberate: the raise has to track the
current largest request in order to be able to drop back, and caching it
would freeze a ceiling that nothing would ever lower again. The set of
raised namespaces is small.

The log moves from Error to Info. Nothing is wrong when this fires; the
operator is reporting a ceiling it chose.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:44 +03:00
Myasnikov Daniil c65db1ad52 perf(operator): skip the memory scan when the LimitRange is settled
reconcileSystemDefaultsLimitRange runs for every system namespace on
every Package reconcile, and Package reconciles follow HelmRelease status
churn through Owns, so on a platform with dozens of system namespaces the
pod and workload scan behind the guard is by far the most expensive thing
in the function -- and it is uncached by design, because caching it would
mean a cluster-wide Pod informer in an operator whose job here is to
bound memory.

Build the desired LimitRange first, read the existing one from the
informer cache, and return without scanning when the two specs already
match. That is safe because the LimitRange being applied is itself the
guarantee: from that moment LimitRanger rejects any new pod whose memory
request exceeds this default, so the only requests a scan could still
find are ones that predate the LimitRange, on pods already running, and
there is nothing it would do differently about them. Steady state becomes
one cached GET.

The comparison is apiequality.Semantic, not reflect.DeepEqual, because
resource.Quantity carries its parsed format and cached string: 4Gi and
4294967296 are the same ceiling and must not read as drift, or every
namespace would scan and re-apply forever.

The test counts List calls rather than only checking the resulting
LimitRange, which cannot tell a skipped scan from a repeated one that
reached the same answer, and seeds an 8Gi pod so a scan that ran would
also change the object.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:44 +03:00
Myasnikov Daniil b55c4f347a docs(operator): keep the residual-gap note on the request scan
The pod-only scan documented what it could not see: a request that only
ever exists post-admission, on a workload whose first pod is rejected
outright. Reading pod templates narrows that gap rather than closing it,
and the rewritten comment dropped the note entirely, which reads as a
claim that nothing is left.

Restore it in its narrower form: a template under the limit, a webhook
that raises the request past it, and a first pod rejected before it
exists, is still invisible to this scan.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:43 +03:00
Myasnikov Daniil 10acfe025d test(operator): cover both terminal pod phases in the skip
findRequestAboveDefaultLimit skips a pod in PodSucceeded or PodFailed,
because neither is ever recreated from that spec. Only the Succeeded half
had a fixture, so deleting the PodFailed clause left the whole ./internal
suite green -- half the condition was uncovered.

Give the case a Failed pod as well. Deleting either clause now fails it.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:43 +03:00
Myasnikov Daniil 654412acd2 fix(operator): scan workload templates for oversized memory requests
reconcileSystemDefaultsLimitRange withholds the LimitRange when a
container in the namespace requests more memory than the configured
default limit, because LimitRanger would then default a limit below that
request and the API server rejects the pod. The scan behind that decision
listed pods only, so it could not see a workload that currently has no
pods: a Deployment or StatefulSet at replicas: 0, a CronJob between runs,
a Job with spec.suspend. Those are ordinary steady states rather than
corner cases. The namespace looked safe, the LimitRange was applied, and
the pod was rejected at the next scale-up or the next schedule with "must
be less than or equal to memory limit of 4Gi" -- a workload that cannot
come back, discovered at the moment somebody needs it.

Scan the pod templates of Deployments, StatefulSets, DaemonSets, CronJobs
and Jobs as well as live pods. Both halves are needed: a template is the
only place a scaled-to-zero workload appears at all, and a live pod is
the only place a request that the VPA admission webhook raised after
admission appears. ReplicaSets are deliberately left out -- a live one
copies its Deployment's template, and the older revisions a Deployment
keeps carry superseded templates that only an explicit rollout undo would
ever instantiate, so scanning them would withhold the LimitRange over a
request replaced releases ago. Jobs are named separately because not
every Job has an owner whose template would cover it, and a finished one
is skipped for the same reason a terminated pod is. A suspended CronJob
is not skipped: suspend is the same dormancy as replicas: 0, undone by
the same kind of human action, and exempting it would leave the gap open
for one kind.

Templates are scanned before pods and the comparison stays strict, so a
workload and its own pods report once, as the workload -- the object an
operator has to edit, and one that outlives pod names. A live pod
outranks its own template only when its request is strictly larger, which
is exactly the webhook-injected case. memoryRequestBlocker therefore
carries Kind/name instead of a bare pod name, and the log key is
"workload" rather than "pod".

Every added List fails closed, which the caller already relies on: a scan
that cannot read is not evidence that applying is safe, so the namespace
is skipped and logged instead. The test for that seeds a sentinel
LimitRange, because asserting the object still exists cannot tell "left
alone" from "applied blind" -- a swallowed List error applies a fresh
LimitRange that exists just the same.

RBAC needs no chart change: the operator's ServiceAccount is bound to
cluster-admin, so it can already list these kinds. The kubebuilder
markers are extended anyway to record what the controller reads.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:42 +03:00
Myasnikov Daniil 7cc090b05f test(operator): pin the largest-offender scan to all three fields
The "largest offender is reported" case asserted only the container
name, and systemPod named every container "app", so the assertion held
whichever of the two pods findRequestAboveDefaultLimit had picked. The
comparison that keeps the largest request was uncovered.

Name the fixture container after its pod, assert the reported pod,
container and quantity together, and give the case three offenders whose
largest sits in the middle of the name-ordered pod listing, so keeping
the first, the last or the smallest offender all name a different pod
and quantity. The init-container case now pins the same three fields.

No production behaviour changes.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:42 +03:00
Myasnikov Daniil 546d8e6a54 feat(operator): default system namespace memory via LimitRange
The Talos userspace OOM handler (v1.12+) picks its victim by ranking
cgroups with `memory_max.hasValue() ? 0.0 : {Besteffort: 1.0, Burstable:
0.5, ...}[class] * memory_current`, discarding every cgroup that scores
zero. Only a memory limit sets memory.max on the pod cgroup, so only a
limit takes a pod out of the candidate set: a request merely demotes
BestEffort to Burstable, which under the v1.13.6 default
strictCgroupClassOrdering still means "killed second". PriorityClass has
no effect at all, because the handler walks cgroupfs and never talks to
the API server.

Victim selection is decoupled from the trigger, so the cgroup that
caused the pressure and the cgroup that gets SIGKILLed are unrelated by
design. Tenant workloads generally carry limits and system components
generally did not, which inverted the intended order: a tenant workload
thrashing against its own limit drives node-wide memory PSI, and
metallb-speaker or linstor-satellite is killed for it while the workload
responsible is never touched.

An audit found roughly 70 workloads across ~30 packages with at least
one container missing a memory limit, so per-chart resources would
neither cover the current set nor keep covering it. Maintain a
LimitRange in every system namespace instead, at the single choke point
that already creates them and already distinguishes system from tenant
(reconcileNamespaces). Tenant namespaces are left alone because
packages/apps/tenant owns LimitRange policy there.

The limit is a ceiling, not a reservation: it defaults well above real
usage (4Gi against a largest observed system request of 512Mi).
defaultRequest is set explicitly and small, because Kubernetes otherwise
defaults each request to the limit and reserves that much at schedule
time. Setting the limit to 0 disables the LimitRange and removes any the
operator previously created.

Two things keep the feature from being able to hurt more than it helps.

A LimitRange default only reaches a container that declares no memory
limit of its own, and the API server then rejects a request above its
limit. So before applying, scan the namespace for a container that
requests more than the configured limit and sets none, and where one
exists skip the namespace, retract any LimitRange applied earlier, and
log the pod, container, request and limit. This is not hypothetical:
packages/system/monitoring and packages/system/monitoring-agents ship
VPAs for cozy-monitoring with maxAllowed.memory of 8Gi and 6G under
updateMode: Initial, so the VPA admission webhook -- which runs after
the in-tree LimitRanger -- can write a request past 4Gi that no chart in
the tree declares. That is why the check runs against live pods rather
than rendered templates, and why it reads through APIReader rather than
starting a cluster-wide Pod informer.

And no failure inside the feature propagates: a LimitRange error, or a
pod list that cannot be read, is logged and stepped over instead of
failing the Package reconcile. A namespace without a LimitRange is back
to the behaviour of every earlier release, while a namespace that never
reconciles leaves its components uninstalled.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:41 +03:00
Aleksei Sviridkin d2e3eb4f82 fix(kubernetes, e2e): unblock autoscaling below min size, size the computeplane pool, fix the backupcontroller build (#4768)
## What this PR does

Makes the backupcontroller test package compile again on main. #4587
added a flavor argument to `createCNPGBackupArtifact` and
`marshalCNPGBackupSnapshot`, and #4669, whose CI ran before that, added
two test calls with the old argument lists. Both PRs were green on their
own, and together they break `go test ./internal/backupcontroller/`. The
two calls now pass the same default flavor as the other tests in the
file.

The second commit fixes an e2e suite that fails the required E2E check
here. It is the `computeplane` suite from #4713, and its worker pool was
one node with a single vCPU. After the kubelet reservations that node
has 900m of CPU to allocate. The tenant addons (cilium, coredns, csi,
ingress-nginx, metrics-server, cert-manager) request exactly 900m
between them. If cert-manager's startupapicheck hook pod is created
last, it can't be scheduled and the install times out after 10m. The
reinstall after remediation runs into the same full node. Whether that
pod comes last depends on install order, so the suite passed on #4713
and failed here.

The pool now has two workers with one vCPU each, the same shape the
kubernetes suites use. One vCPU per worker stays on purpose, because of
#3513.

Two workers exposed a deadlock in the tenant cluster-autoscaler. The
suite's pool is created at zero replicas and then gets a min size of 2,
which CAPI does not apply to the live replica count on upgrade (#4786).
The autoscaler's scale-up for a pending pod asks for one node, and the
clusterapi provider refuses it because one is under the min size,
although it is an increase. That refusal backs the group off, and the
min-size enforcement skips groups in backoff, so the pool never leaves
zero. Upstream had the same report in kubernetes/autoscaler#4942, closed
as stale, and it is now tracked in kubernetes/autoscaler#10400. Our
autoscaler image already patches the mirror case for max size, and the
patch now refuses only a decrease below the min size.

`go vet` and `go test ./internal/backupcontroller/...` pass, and so do
the bats pins for the computeplane suite. The autoscaler patch adds a
test for each direction, and the clusterapi package tests pass with it
applied to 1.32.0.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved cluster autoscaling for worker groups outside configured size
limits: groups below their minimum can scale up toward it, and nodes can
be removed when a group exceeds its maximum. Scale-down requests below
the minimum remain restricted.
* **Tests**
* Updated backup and restore test calls with PostgreSQL and empty flavor
values.
* Updated ComputePlane checks for a two-worker pool and clarified
cert-manager installation timing expectations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 04:36:10 +03:00
Aleksei Sviridkin ff3c409881 fix(backups): refuse a foreign -orig PVC and record the kept disk's reclaim policy (#4671)
## What this PR does

Two defects in `renamePVC`
(`internal/backupcontroller/velerostrategy_controller.go`), the step of
an in-place VMInstance restore that keeps each disk as
`<dv>-orig-<shortHash(RestoreJob name)>` when `keepOriginalPVC` is true,
the default.

### A taken `-orig` name skipped the rename, and the restore went on

`renamePVC` returned `nil` as soon as a PVC with the `-orig` name
existed (`main:1263-1266`), whatever that PVC was. `prepareForRestore`
then reached Step 4 (`main:1172-1183`), which deletes the DataVolume of
every disk regardless of what Step 3 did, because a Step 3 error was
only ever turned into a Warning event (`main:1165-1168`). The restore
therefore ran exactly as `keepOriginalPVC: false` does and reported
`Succeeded`: the live disk was not kept, and the `-orig` name held a
different, older disk that the user takes for the pre-restore copy.

When that name can already exist:

- A RestoreJob created under the name of a deleted one. The hash covers
only the RestoreJob name, and nothing in the controller ever deletes an
`-orig` PVC, so the predecessor's PVC is still there. This is the
realistic case, and it is what a scripted restore with a fixed
RestoreJob name runs into on its second run unless the `-orig` PVC was
removed in between.
- Two RestoreJobs with different names whose 4-hex-digit hashes collide
(16 bits) while restoring the same disk in the same namespace. Rare, but
nothing guards it.
- A reconcile of the same RestoreJob after a controller restart in the
middle of a rename. The name is ours, but the early return treated a
half-done rename as complete: the original PVC stayed in place and the
PV stayed bound to it.
- Truncation does not apply: the name is not truncated, and one longer
than 253 characters makes the `Create` fail, an error that was swallowed
in the same way.

Each `-orig` PVC now carries the UID of the RestoreJob that created it,
in `backups.cozystack.io/restore-job-uid`. `prepareForRestore` checks
every `-orig` name before it suspends or halts anything, and fails the
RestoreJob, naming the PVC, when any of them is held by a PVC another
RestoreJob created. When the PVC is ours, `renamePVC` resumes: it
deletes the original PVC only while it still claims the kept PV, and
repoints the PV unless it already points at the `-orig` PVC, so a rerun
is a no-op. Any other rename error now holds the restore back before
Step 4 deletes a DataVolume, and the restore is retried, instead of the
error becoming an event (see the next section).

An `-orig` PVC created by an earlier controller has no UID annotation
and is refused like any foreign PVC. The only RestoreJob that loses by
this is one whose controller was upgraded after its rename and before
its Velero Restore was created; it fails with the message above rather
than going on.

### The rename read its own write back through the cache

`renamePVC` created the `-orig` PVC, then read it back through the
cached client to learn its UID for the PV's claimRef (`main:1314-1316`).
The informer often has not seen a PVC created a moment earlier. On our
lab, running Cozystack v1.6.3, an in-place restore of a two-disk VM with
`keepOriginalPVC: true` renamed and restored the system disk, while the
data disk lost the race, with this event on the RestoreJob: `Failed to
keep old PVC vm-disk-e2e-dc-fix-data: failed to get -orig PVC
vm-disk-e2e-dc-fix-data-orig-a574: PersistentVolumeClaim
"vm-disk-e2e-dc-fix-data-orig-a574" not found`. As the error was only an
event, the data disk's DataVolume was deleted and its original PVC
stayed in place. Velero skipped that PVC and created no DataDownload for
it, so its content was not restored, yet the restore finished
`Succeeded`. The `-orig` PVC was left `Pending` on the same PV, which
was by then `Retain`.

`Create` already fills in the UID, so the read-back is gone. The other
reads the rename decides on (the `-orig` PVC, the original PVC and the
PV, before and after its own writes) go through the manager's uncached
reader, wired as `APIReader`. The PV rebind retries on conflict, since
the PV controller also writes the PV once the original PVC is deleted. A
rename that fails for any reason other than a foreign `-orig` name
requeues the restore rather than failing it. The DataVolume is still
there, and the retry finds its own `-orig` PVC by the RestoreJob UID and
resumes where it stopped. A cause that never clears keeps the RestoreJob
`Running`, with a Warning event on each attempt.

#4673 adds the same `APIReader` field and `apiReader()` helper to
`RestoreJobReconciler`, for its finished-job guard. Whichever of the two
lands second drops its own copy when it rebases.

### The PV's reclaim policy was overwritten without being recorded

`renamePVC` switches the PV to `Retain` so it survives the deletion of
the original PVC (`main:1289-1293`), and the policy it had was lost. I
looked for a path in the controller that releases or cleans up `-orig`
PVCs, and there is none: `cleanupOnDelete` on a Velero RestoreJob
removes only the Velero `Restore` and the resource-modifier ConfigMaps,
and no code deletes an `-orig` PVC or touches its PV again. A PV that
was `Delete` therefore stays `Retain` for good, and its volume leaks
when an operator deletes the `-orig` PVC.

The original policy is now recorded in
`backups.cozystack.io/original-reclaim-policy`, on the PV and on the
`-orig` PVC, following the `backups.cozystack.io/` prefix the controller
already uses for `underlying-resources`. A value already present on the
PV is kept, as it predates any rename. Since there is no release path to
restore it from, `examples/backups/vmi/scenario-user-restore.md` gains a
section for the operator who deletes an `-orig` PVC: set the recorded
policy back on the PV first, with a snippet that leaves a PV without the
annotation on `Retain`.

Restoring the policy from the controller itself, for instance once the
Velero Restore has completed and the PV is `Bound` to the `-orig` PVC,
would close the leak without the operator. I did not do it, because it
changes what deleting an `-orig` PVC destroys, from nothing to the
volume; that is a maintainer call, and the annotation is what it would
read.

### How this fits with #3427

#3427 is complementary, and the two do not touch the same code: it
changes `collectUnderlyingResources`, this PR changes
`prepareForRestore` and `renamePVC`. Both append tests to
`velerostrategy_controller_test.go`, so whichever lands second may need
a trivial rebase.

- #3427 makes a standalone VMDisk report its own disk, which sends
VMDisk in-place restores through this same `renamePVC`. Once both land,
the VMDisk path gets the same guard and the same annotation.
- Its `examples/backups/{vmi,vmdisk}/cleanup.sh` and the Chainsaw
`finally` reclaim each `-orig` PVC by patching its PV to `Delete` before
deleting the PVC. That works around, in the e2e teardown, the leak
described above. It is correct on the LINSTOR classes, which are
`Delete`, but it would destroy a volume whose class was `Retain`. With
this PR the scripts can read
`backups.cozystack.io/original-reclaim-policy` instead of hard-coding
`Delete`. I have not changed them here, since they only exist on that
branch.
- Its round-trips use fixed RestoreJob names and assert that a
`<pvc>-orig-*` PVC exists after the restore. If a failed teardown leaves
one behind, today the next run skips the rename and the assertion still
passes on the stale PVC; with this PR the RestoreJob fails and names it.
- Its cleanup comments describe the `-orig` PVC as a bare ObjectMeta
with no labels or owner. That stays true: this PR adds annotations only,
so its name-based lookup is unaffected.

### Tests

New tests in
`internal/backupcontroller/velerostrategy_controller_test.go`, run
through `prepareForRestore` with the DataVolume in the dynamic fake
client:

- `TestPrepareForRestore_KeepOriginalPVC_RefusesForeignOrigPVC`: an
`-orig` PVC without the annotation, and one carrying another
RestoreJob's UID, both fail the restore with the PVC named, and leave
the DataVolume, the live PVC, the PV and the foreign PVC untouched.
- `TestPrepareForRestore_KeepOriginalPVC_ResumesOwnRename`: a rename
interrupted after the `-orig` PVC was created completes on the next
pass, and a second pass changes nothing.
- `TestPrepareForRestore_KeepOriginalPVC_FailedRenameKeepsDataVolume`: a
rename that fails for another reason (missing PV) requeues the restore
without failing it and keeps the DataVolume, and once the PV exists the
retry completes the rename.
- `TestPrepareForRestore_KeepOriginalPVC_CacheLagsBehindWrites`: the
reconciler's cached client serves every read from the objects as they
were seeded, through an `interceptor` over the fake client, as an
informer that has not caught up with this reconcile's writes would,
while `APIReader` sees them. The rename must complete.
- `TestRenamePVC_RecordsOriginalReclaimPolicy`: a `Delete` PV and a
`Retain` PV both end up `Retain`, with their original policy recorded on
the PV and on the `-orig` PVC.

With the controller reverted to `main` (only the two annotation
constants kept so the tests compile), all four fail:
`prepareForRestore() = ready true, nil error` with `DataVolume was
deleted although the disk could not be kept` for both foreign cases and
for the failed rename, `original PVC still exists` and the PV still
bound to it for the interrupted rename, and an empty
`original-reclaim-policy` on the PV and on the PVC. For the cache race,
the first commit alone fails `CacheLagsBehindWrites` with the lab's
error, `failed to keep PVC vm-disk-ubuntu-source: failed to get -orig
PVC vm-disk-ubuntu-source-orig-de3e: persistentvolumeclaims
"vm-disk-ubuntu-source-orig-de3e" not found`, and
`FailedRenameKeepsDataVolume` with a terminal error instead of a
requeue. On `main` the race test reproduces the lab outcome: `original
PVC still exists`, and the PV still claimed by it.

With both commits, `go test -count=1 ./internal/backupcontroller/...`
passes.

No API type, CRD, chart or values file changes, so there is nothing to
regenerate.

### Screenshots

N/A, no UI change.

### Downstream repositories

Walked the trigger map against the diff: the restore preparation in
`internal/backupcontroller/velerostrategy_controller.go`, its tests, and
`examples/backups/vmi/scenario-user-restore.md`. No CRD,
ApplicationDefinition, values, schema, enum, namespace, label, metric or
tooling change, so terraform-provider-cozystack, ansible-cozystack, ccp,
talm, cozyhr, cozy-proxy, cozystack-telemetry-server,
external-apps-example, examples and community are not reached.
`RestoreJob.spec.options` stays opaque to the provider.

cozystack/website is reached:
`content/en/docs/next/virtualization/backup-and-recovery.md` (and each
released copy) tells users to delete the `*-orig-*` PVCs "to free up
storage". A PV left on `Retain` frees nothing, and the page should
explain the recorded policy and the new failure for a reused RestoreJob
name. No follow-up exists there yet, so the box stays empty until one is
filed and linked.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): an in-place VM restore with keepOriginalPVC now fails, before the VM is halted, when the <disk>-orig-<hash> PVC it would keep the disk under already exists and was not created by this RestoreJob (for instance one left by a deleted RestoreJob of the same name), instead of skipping the rename and restoring without keeping the current disk. Any other failure to keep a disk, including a race in which the rename read its just-created -orig PVC back through the cache and failed, now holds the restore back and retries it before its DataVolume is deleted. Previously such a restore reported success without restoring the disk. The PV's reclaim policy before the switch to Retain is recorded in the backups.cozystack.io/original-reclaim-policy annotation on the PV and the -orig PVC; set it back on the PV before deleting the -orig PVC, or its volume is not reclaimed.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* In-place restores that preserve original PVCs can now resume safely
after interruptions, including when reads lag behind recent changes.
* Restores stop before suspending the VM if a conflicting `-orig` PVC
already exists. PVC rename failures can be retried, while conflicting
names remain fatal.
* Original PV reclaim policies are recorded and restored before
preserved PVCs are deleted.
* **Documentation**
* Updated the restore guide with disk-reclamation steps and guidance for
resolving leftover PVC name conflicts.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 16:51:49 +03:00
Aleksei Sviridkin ae738e6047 test(backups): pass the flavor in the major-version test calls
Two tests that came with the PostgreSQL major-version check call
createCNPGBackupArtifact and marshalCNPGBackupSnapshot with the
argument lists from before the flavor parameter was added, so the
backupcontroller test package does not compile on main. Both calls
take the default flavor the other tests in the file pass.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-06 16:39:12 +03:00
Aleksei Sviridkin 3fc45ed183 fix(backups, postgres): refuse a restore into another PostgreSQL major (#4669)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A CNPG restore is physical, so the recovered data files only open on the
PostgreSQL major version that wrote them. `reconcileCNPGRestore` purged
the target Cluster and its PVCs without comparing the two. Observed on
v1.6.3: a PostgreSQL 18 backup restored into a v16 application emptied
the target, the `full-recovery` pods then looped on `FATAL: database
files are incompatible with server`, and the RestoreJob failed at the
30-minute deadline with the target left empty. An in-place restore into
an application whose `spec.version` was raised since the backup ends the
same way.

The restore now compares the two majors before the purge, ahead of the
WAL-archive gate, and when they differ marks the RestoreJob `Failed`
with reason `MajorVersionMismatch` and a message naming both versions.
Nothing is suspended, patched or deleted on that path.

- Target major: the Postgres application's `spec.version` (`v13` to
`v18`). That is what the chart renders the recovered Cluster from once
the live one is gone, so the live Cluster's image would be the wrong
thing to compare.
- Backup major: the `cnpg.io/Backup`'s `status.majorVersion`, now copied
into the artifact's `driverMetadata` as `cnpg.io/major-version` at
backup time, because barman retention deletes the `cnpg.io/Backup` while
the artifact stays restorable. An artifact taken before this change
falls back to the `cnpg.io/Backup` while it still exists.
- When either side is unknown (no record and the `cnpg.io/Backup` gone,
a CNPG that predates `majorVersion`, an empty `spec.version`), the
restore goes ahead as before and the controller logs it, so existing
backups keep restoring.

`TestReconcileCNPGRestore_RefusesCrossMajorRestore` drives the real
reconcile against a fake client with a live Cluster and PVC on the
target and counts every `Delete`. With the check removed, the two
refusal cases (18 into a v16 copy, 16 in place into an app moved to v18)
and the fallback case fail: the Cluster and the PVC are deleted and the
app is patched for recovery. With the check in place they end `Failed`
with zero deletes, while the same-major and unknown-version cases still
reach the purge. `TestCreateCNPGBackupArtifact_RecordsMajorVersion`
covers the `driverMetadata` record.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Walked against the diff: it touches only `internal/backupcontroller`,
with no CRD, no app `values.yaml` or schema and no ApplicationDefinition
change. The new `cnpg.io/major-version` entry lives in the free-form
`Backup.spec.driverMetadata` map, which no downstream repository reads
by key.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backups, postgres): a Postgres restore whose backup was taken on another PostgreSQL major version than the target's spec.version now fails before touching the target, instead of purging it and then failing to recover.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Cross-major PostgreSQL restores are now refused before the target is
changed. The error identifies the backup and target versions.
* Restores proceed when either version is unavailable or cannot be
parsed.
* **New Features**
* Backup metadata now records the PostgreSQL major version when it is
available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 16:35:27 +03:00
Aleksei Sviridkin 9734a16692 feat(postgres): add a postgis image flavor (#4587)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Adds a `flavor: postgresql | postgis` enum to the `postgres` app.
`postgis` runs the cluster on the CloudNativePG PostGIS operand image,
so tenants can enable `postgis`, `postgis_raster`, `postgis_topology`,
`postgis_sfcgal`, `postgis_tiger_geocoder` and `address_standardizer`
through the existing `databases.<name>.extensions`. The default stays
`postgresql` and renders exactly the image it rendered before this
change.

This picks up #2672 by @matthieu-robin, which the stale bot closed after
a rebase request, with these changes:

- The postgis tags in #2672 (`17.7-3.5`, `16.11-3.5`, ...) do not exist
in `ghcr.io/cloudnative-pg/postgis`; the registry answers 404, so the
flavor would have ended in `ImagePullBackOff`. The map now uses tags
that exist, `<minor>-<postgis>-standard-trixie`, on the same PostgreSQL
minor as `files/versions.yaml`.
- v18 is supported: PostGIS images for 18 now exist upstream.
- `standard` rather than `system`: the chart archives through the
barman-cloud plugin since the backup rework, so the operand no longer
needs the barman binaries of the deprecated `system` variant.
- `hack/update-versions.sh` derives the postgis map from the minors it
picks, and `hack/check-postgres-postgis-versions.bats` fails if the two
maps drift to different minors or away from `standard-trixie`.
- The chart refuses a flavor change on a live `Cluster`. The default
images are built on Debian bullseye (glibc 2.31) and the postgis ones on
trixie (glibc 2.41): CNPG would roll the swap out as a plain image
update, under an initialised data directory, which silently invalidates
every index on collatable text, and the postgis-to-postgresql direction
also drops the libraries behind every PostGIS object. Only the image
name is compared, so minor bumps and registry mirrors keep passing.
- `pgrouting` is not claimed: it is absent from the `18.1-3.6.1` build
the chart pins and only appears in later postgis builds.

A general mechanism (CNPG ImageVolume extensions) was considered and
left out: it needs PostgreSQL 18 and Kubernetes 1.35 or the
`ImageVolume` gate, so it cannot serve v13-v17 nor most management
clusters today, and PostGIS is the only extension CloudNativePG ships as
a full operand image. No other extension request exists in this
repository.

Testing:

- `helm unittest packages/apps/postgres`: 145/145, including 6 new cases
in `tests/flavor_test.yaml` (default image, postgis on the Cluster and
on the init Job, the guard in both directions with a mocked live
Cluster, and a mirrored digest-pinned image taking a new minor).
Removing the guard makes exactly the two guard cases fail.
- `hack/cozytest.sh hack/check-postgres-postgis-versions.bats` passes,
and fails on a shifted minor or a `system` tag.
- `make -C packages/apps/postgres generate` leaves no diff.
- The rendered
`ghcr.io/cloudnative-pg/postgis:18.1-3.6.1-standard-trixie` was started
locally: `CREATE EXTENSION` for citext, postgis, postgis_raster and
postgis_topology succeeds and `ST_Buffer` answers. A three-instance
cluster on the `system` sibling of that image has been running PostGIS
in production since 2026-09-29.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: cozystack/terraform-provider-cozystack#63 (issue: the
provider models every field by hand, and the default handling needs a
decision there)
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(postgres): add a `flavor` field accepting `postgresql` (default) or `postgis`. `postgis` bootstraps the cluster from the CloudNativePG PostGIS image, so PostGIS extensions can be enabled per database through `databases.<name>.extensions`. The flavor is fixed at creation: changing it on an existing cluster is refused, and so is restoring a backup into a cluster of the other flavor, whether through a RestoreJob or `bootstrap.enabled`; move data across flavors with a logical dump.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added a `flavor` option for PostgreSQL clusters. The default
`postgresql` flavor remains available, while `postgis` selects an image
with PostGIS extensions.
  - PostGIS images are available for PostgreSQL versions 13–18.
- Cluster flavor is fixed at creation. To switch flavors, create a new
cluster and migrate data with a logical dump.
- **Bug Fixes**
- Restores are refused when the backup and target cluster use different
flavors.
- **Documentation**
- Updated PostgreSQL configuration guidance to describe flavor options
and limitations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 16:17:22 +03:00
Aleksei Sviridkin f527ef17ee test(fluxcontract): accept a timeout-bounded history read
The monitoring suite bounds its HelmRelease history read with
`timeout 30 kubectl`, which keeps a hung API call from eating the
step's whole budget and the diagnostic with it. The shared-read pin
admitted kubectl only bare or through the bounded wrapper, so the two
landed side by side and left the unit tests red on main.

The pin now takes an optional `timeout [-k N] N` in front of kubectl,
the forms the suites already use for bounded reads. The bound is
spelled out instead of allowing any prefix, since
`$(echo kubectl ...)` would name the expansion without running it;
that form, bare and behind a timeout, is now a refuse case. So is a
timeout in front of _cozy_guard_kubectl: timeout runs a program, the
wrapper is a shell function, and the read would fail every time. The
command name has to end at a blank, so `kubectlx` or
`_cozy_guard_kubectl_v2` is not taken for either one.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-05 17:57:04 +03:00
Aleksei Sviridkin 415430bcc7 fix(e2e): fail on a failed release of cilium and csi as on the parent, and report addon remediation (#3594)
## What this PR does

The history guard in the tenant Kubernetes e2e suite checked only the
parent `kubernetes-<test>` HelmRelease. A failed action on one of the
addon releases it installs left nothing the suite looked at, so a run
showed only the downstream symptom, usually the node-join deadline, and
never named the release behind it.

The guard now reads every addon release of the parent. It selects them
from the cluster by name prefix, so a release added to the chart later
is covered too. The rule:

- A `failed` or `uninstalled` Snapshot fails the run on any release
whose install strategy is `RetryOnFailure`. That is the parent (the
aggregated apiserver stamps it on every Application HelmRelease) and the
tenant `cilium` and `csi`.
- The same footprint on the other addons, which run the default strategy
with `retries: -1`, is printed as a note and the run stays green. For
the CRD addons that configuration is tracked in #3555.

A transient `cilium` or `csi` failure that later recovers on a retry
still turns the run red, by design, as long as its Snapshot still reads
`failed`. One recovery leaves no such Snapshot: when an install fails
and the very next attempt succeeds, Helm marks the failed install
`superseded`, because there is no deployed release to supersede, and the
history reads like an ordinary upgrade. The guard cannot see that case
from the statuses alone. The code and the docs now say so, a test pins
it, and detection is left to a follow-up.

An earlier revision of this PR tried to tell a teardown apart from a
retried failure and made only `uninstalled` fatal. That split does not
hold. In helm-controller v1.5.0, the version the platform installs, a
release with an active retry strategy never remediates.
`atomic_release.go` returns an upgrade for a failed release before it
reaches the remediation switch, and the two other paths that uninstall
(a changed release target, and deletion) both clear history afterwards.
So on these releases `uninstalled` never shows up, and `failed` is the
only signal. The parent keeps the rule it already had on main, and
`cilium` and `csi` now follow it too.

The failure paths get the same reading. The guard runs after the
assertions, while the deadlines a failed addon breaks end the script
earlier, so the same classification is also printed, without a verdict,
from the node-join collector and from the exit handler. That handler is
now armed at the tenant CR apply, so the management-cluster waits before
the tenant answers are covered too. Smaller changes that came with this:

- Every guard read is bounded, and a failed read is never folded into
"no history".
- The parent must read back a non-empty strategy. An empty value there
means the jsonpath stopped resolving, and without this check every
footprint would quietly turn into a note.
- The history read uses `HELMRELEASE_HISTORY_JSONPATH`, and the
upstream-type pin in `internal/fluxcontract` now accepts the bounded
wrapper the read goes through.

`docs/agents/e2e-testing.md` §6 and checklist item 8 describe the new
rule.

On related issues: this adds the `local` that #3774 reports missing in
`helmrelease_has_remediation_cycle`, but not the sweep over the other
`_lib/` helpers that the issue also asks for. For #3773 it rewrites the
guard comment the issue quotes. The waits without a backstop are not
touched.

Closes #3553.

### Screenshots

Not a UI change.

### Downstream repositories

I walked the trigger map against the diff. It touches
`hack/e2e-chainsaw/_lib/`, `hack/*.bats`, one Go test under
`internal/fluxcontract`, two spend-order comments in the kubernetes
suites, and `docs/agents/e2e-testing.md`. No package, schema, CRD, make
target or user-facing document changes.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(e2e): the HelmRelease history guard now also covers the tenant addon releases. A failed or uninstalled Snapshot fails the run on the parent, cilium and csi, which retry in place; other addons get a note. The same classification is printed on the failure paths where the guard itself cannot run.
```
2026-10-05 17:12:38 +03:00
IvanHunters b0e07cadbb fix(operator): rebuild each changed ArtifactGenerator generation once (#4594)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

After an upgrade, an ArtifactGenerator can report Ready on its new spec
while its artifacts were built from the old one, and source-watcher
never rebuilds them. That is how the bucket-system chart in #4593
shipped without cozy-lib.

The cause is upstream (fluxcd/source-watcher#384). source-watcher builds
from the spec it read when the reconcile started, but the final status
patch stamps Ready with the live generation. An upgrade changes the
packages source and the copy rules within seconds, so a spec change
often lands during a build.

The generator status has nothing that tells a stale build from a good
one. So the operator now rebuilds every changed generation once, after
source-watcher first reports it Ready, through the force-drift path the
operator already uses for fluxcd/pkg#934. An unchanged spec rebuilds to
the same digest, so ExternalArtifacts and HelmReleases do not move.
Generation 1 is skipped, since the race needs a spec change during a
build.

On the first rollout every generator past generation 1 gets one extra
rebuild, and its PackageSource shows
`Unknown/AwaitingSourceWatcherRecovery` until that rebuild lands.
Nothing waits on that condition today. The same rebuild repairs clusters
already stuck like #4593.

The recovery messages on the PackageSource no longer blame
fluxcd/pkg#934, because the same path now serves both cases.

### How it was tested

Unit tests in `internal/operator/packagesource_reconciler_test.go` cover
the forced rebuild on generation 2 and on a later generation, every case
where no rebuild must happen, the rebuild settling without a second one,
both failure orders (force fails, recording the generation fails), and a
stale recovery counter. Each condition of the new predicate was mutated
once to confirm a test fails.

I also ran this commit's operator image on an arm64 test cluster running
v1.7.0-alpha.3, the one where #4593 was found. Before that I had
repaired its bucket generator by hand: deleting the stale
ExternalArtifact and requesting a reconcile produced an artifact with
`charts/cozy-lib` and both bucket HelmReleases went Ready. With the
patched operator, the only generator past generation 1 got exactly one
forced rebuild and the verified-generation annotation. source-watcher
reported all its ExternalArtifacts up to date. The digests of all 214
ExternalArtifacts and the revisions of all 104 HelmReleases matched a
snapshot taken before the swap, and every PackageSource and HelmRelease
stayed Ready. The race itself was not reproduced on the cluster, since
that needs a spec change landing during a build.

Closes #4593

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(operator): rebuild an ArtifactGenerator once after each spec change, so an upgrade no longer leaves a component on artifacts built from the previous spec (for example a chart missing a library it now needs).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved PackageSource recovery when an artifact generator reports
ready: the system now verifies the generated output and triggers a
rebuild when needed.
* Failed rebuild or verification attempts can be retried during a later
reconciliation, and recovery tracking is cleared once the rebuilt
generation settles.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:43:44 +03:00
Mattia Eleuteri cfa31d3a17 fix(postgres): record the live cluster's flavor on a backup
The chart refuses a flavor change on an existing Cluster, but the
Postgres app keeps the refused value. A BackupJob taken after such a
change recorded the app's flavor, not the one the data directory was
written by, so a RestoreJob would accept a target of the wrong image
family and refuse the right one.

The driver already reads the live Cluster to keep its barman
serverName; it now takes the flavor from that Cluster's imageName the
same way the chart's flavor guard compares images.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-05 08:52:32 +02:00
Aleksei Sviridkin 2df578bffd test(e2e): read the opensearch release history through the pinned name
The opensearch suite wrote the history jsonpath out as a literal, so
the contract test that runs HELMRELEASE_HISTORY_JSONPATH against the
upstream Flux type never saw the expression this suite hands kubectl.
A field rename upstream would have reached it only as an empty read.

The suite now takes the shared expression, and the contract test reads
every Chainsaw suite that calls the guard, so a suite copying the
literal, or assigning the shared name after sourcing the library, fails
in the unit lane instead of going unchecked. The read is matched inside
the script step that calls the guard, because every step runs in a
shell of its own and a read in another step never reaches the call.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 11:39:34 +03:00
Aleksei Sviridkin 8cb1023788 fix(e2e): fail on a failed release of cilium and csi as on the parent
The history guard covered only the parent Kubernetes HelmRelease. A
failed action on one of the addon releases it installs left no mark
the suite inspected, so a run surfaced only the downstream symptom,
typically the node-join deadline, with nothing naming the release
behind it. The tenant CNI and CSI are the sharp cases, since they
carry a cluster-wide precondition.

The guard now reads every addon release of the parent, selected from
the cluster by name prefix so a release added later is covered too.
A "failed" or "uninstalled" Snapshot fails the run on a release whose
install strategy is RetryOnFailure: the parent, which the aggregated
apiserver stamps that way, and cilium and csi. The same footprint on
the other addons, which pair the default strategy with retries: -1,
is printed as a note and leaves the run green.

The footprint is not split into a teardown and a retried failure.
In helm-controller v1.5.0, the version the platform installs, and
unchanged in v1.5.1, a release with an active retry strategy never
remediates: atomic_release.go returns an upgrade for a failed
release before it reaches the remediation switch (lines 465-475), and
the two other paths that uninstall, a changed release target and
deletion, both call ClearHistory afterwards. On a RetryOnFailure
release an "uninstalled" Snapshot is therefore unreachable, and the
"failed" one is the only signal. A retry that later recovered still
fails the run, by design, with one blind spot the statuses cannot
close: a failed install that the very next attempt recovers is
marked superseded by Helm, because there is no deployed release to
supersede instead, and its history reads like an ordinary upgrade.

The guard runs after the assertions, while every deadline a failed
addon breaks ends the script earlier, so the same classification is
also printed without a verdict from the node-join collector and from
the exit handler. That handler is now armed at the tenant CR apply,
so the management-cluster waits before the tenant answers are
covered too. Every read is bounded and never folded into "no
history", and the parent must read back RetryOnFailure: an empty
value there means the jsonpath stopped resolving, and any other value
means the apiserver stamp is gone, either of which would turn its
footprint into a note.

The history read takes its jsonpath from HELMRELEASE_HISTORY_JSONPATH,
and the upstream-type pin accepts the bounded wrapper the read goes
through.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 06:59:31 +03:00
Aleksei Sviridkin 2eeaa59055 test(controller): seed the DataVolume informer for controller-runtime 0.25
controller-runtime 0.25 made FakeInformer's sync state a private
channel, so the DataVolume watch tests no longer compile: they toggled
the exported Synced field and its lock. FakeInformers also builds every
informer it creates already synced, so an unsynced informer has to be
seeded into InformersByGVK and marked synced with Synced() later.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 22:29:03 +03:00
Aleksei Sviridkin bf03c84711 chore(deps): move to the Kubernetes 1.37 libraries without the apimachinery fork (#4473)
## What this PR does

This moves the root module's `k8s.io/*` from v0.35 to v0.37.1 and
controller-runtime from v0.23 to v0.25.1, and drops the
`k8s.io/apimachinery` fork on the way. Supersedes #3001, #4467 and
#4472; those were a stack of the same commits.

The four commits are meant to be read one by one, and each builds and
passes the tests on its own:

1. **Plain watches stop getting the initial-events-end bookmark.** The
apiserver defaults `sendInitialEvents=true` onto any watch with an empty
or `0` resourceVersion and leaves bookmarks off, and our aggregated
storages took that flag alone as a WatchList request. From
`k8s.io/apiserver` v0.36 the watch handler compiled into cozystack-api
panics on that bookmark, so `kubectl get -w` on any cozystack-api
resource would die after the bump. The marker now needs both flags, the
same condition the watch cache uses. There are tests per storage and one
through the real handler chain.
2. **The apimachinery fork goes.** Every apps kind shares the
`Application` Go type, and stock `Scheme.ConvertToVersion` picks the
first kind registered for it. The fork patched that; the apps group now
encodes through a serializer that keeps the kind the object carries. A
handler-chain test covers create, get, list, update, watch, apply and
table for two kinds and fails without it.
3. **MutatingAdmissionPolicy stays out of admission.** From v0.36 the
plugin is on by default and refuses every write until it has synced
`mutatingadmissionpolicies`, which cozystack-api has no RBAC to list and
which clusters before 1.36 do not serve. Without this, every create,
update and delete would fail after the bump. It is also re-applied in
`Complete`, since `--disable-admission-plugins` replaces the list.
4. **The bump itself**, with the regenerated clients, openapi and
strategy CRDs.

Commits 2 and 4 are where the review effort goes; 1 and 3 are small.

Known changes in behaviour:

- Without the fork, the row objects of a Table requested with
`includeObject=Object` carry the first registered app kind. kubectl asks
for that only with `--sort-by`, and it sorts on fields, not on the kind.
Nothing at this layer can fix it, because the handler converts through
`APIGroupInfo.Scheme`, a concrete `*runtime.Scheme`. Admission webhooks
and ValidatingAdmissionPolicy still see the right kind.
- The alpha PodSpec field `workloadRef` became `schedulingGroup`
upstream. Once the new strategy CRDs are applied, the apiserver prunes
`workloadRef` from stored strategies it serves, and their next write
erases it. It only ever worked behind an alpha feature gate.

The branch is rebased onto the Renovate rollup (#4447), with `go mod
tidy` and codegen rerun on top of it.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
chore(deps): cozystack-api and the controllers build against the Kubernetes 1.37 libraries without the apimachinery fork. Plain watches no longer receive the WatchList initial-events-end bookmark, and objects in Table rows requested with includeObject=Object may carry another app's kind.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Application API responses now preserve the requested resource kind
across operations, including lists and watches.
* Backup strategy resources support additional Kubernetes pod settings,
including probe protocols, volume ownership options, and scheduling
groups.
* Backup strategy schemas include updated Kubernetes pod fields and
descriptions.
* **Bug Fixes**
* Initial-event watch bookmarks are sent only when requested; plain
watches continue without an ending bookmark.
* Mutating Admission Policy is disabled by default, including when
admission plugin options are customized.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-02 21:45:51 +03:00
Aleksei Sviridkin 9809b22280 fix(vm-disk): report DataVolume readiness through WorkloadsReady (#4380)
## What this PR does

Nothing on a vm-disk application tells whether CDI ever finished
populating the disk. Its WorkloadMonitor has no pods to count, and
`Ready` follows the Helm release. So an import or clone that never
completes can leave a green application over a disk nobody can use.
cozystack/cozystack#2908 is one way to get there: a clone into a
StorageClass that cannot place a replica where the source data sits. The
expensive part there was the silence, not the stuck clone.

The disk state was in the DataVolume phase all along, and nothing read
it. With this change the WorkloadMonitor in cozystack-controller reads
it. The monitor lists the DataVolumes its selector matches and reports
not operational while any of them is in flight, Failed, Unknown or has
no phase. An upload counts as in flight until it completes. That
surfaces on the application as `WorkloadsReady=False` with reason
`DataVolumeNotReady` and a message that names the DataVolume and its
phase, for example `DataVolume vm-disk-x is ImportInProgress`.

`Ready` still follows the HelmRelease, as for any other kind
(`convertHelmReleaseToApplication` in
`pkg/registry/apps/application/rest.go`), and for a vm-disk it no longer
waits on the DataVolume at all. So the disk state has to be visible
outside `Ready`, and this PR puts it where operators look by default:

- `kubectl get` on any application kind gets a `WORKLOADS` column next
to `READY`. It shows the `WorkloadsReady` status, or `<none>` when the
application has no WorkloadMonitor.
- The console badge turns amber and reads `DataVolumeNotReady` while
`Ready` is True and `WorkloadsReady` is not True for that reason. A
failing `Ready` still wins. The console shows only the disk case: other
kinds need a separate decision, because a stopped VM legitimately has no
pod and reports `WorkloadsReady=False` too.

A Helm readiness gate on the DataVolume would be the obvious
alternative, and it is not used on purpose. helm-controller runs that
wait inside its reconcile, so an importing disk would hold one of the
shard's worker slots for the whole install window, and a disk stuck for
good would keep taking the slot back. cozystack/cozystack#4145 has the
details. The monitor costs no slot, keeps watching after install, and
covers existing disks without anyone writing to them.

Without any gate, the install would still wait on the DataVolume.
helm-controller's default poller falls back to a resource's `Ready`
condition for kinds it has no rule for (`checkReadyCondition` in
fluxcd/cli-utils `pkg/kstatus/status/status.go`), so a vm-disk install
could hold a worker slot for the whole install timeout. The
ApplicationDefinition therefore gets a `healthCheckExprs` entry for
DataVolume with `current: "true"`, the same idiom postgres-rd uses for
its ScaledObject. With it Helm never waits on the DataVolume. The entry
is pinned by a helm-unittest in `packages/system/vm-disk-rd`, which gets
a `test` target for it.

Release settings reach a HelmRelease only when the application is
created or updated, so an existing disk gets the entry on its next
update. The reconciler that syncs definitions onto existing releases
copies chartRef, valuesFrom and labels only. That affects any kind with
release settings, so it is filed as cozystack/cozystack#4409 and not
fixed here. `WorkloadsReady` covers every disk right away, since the
monitor needs no write to the application.

CDI copies the DataVolume labels onto the PVC it creates and makes the
DataVolume its controller, so the monitor also publishes a `pvc-<name>`
Workload for the disk. That Workload reads not operational while its
DataVolume is not ready, even though the PVC is already Bound, so
`kubectl get workloads` agrees with `WorkloadsReady`. While the
DataVolumes cannot be read it keeps the verdict it has, like the
monitor.

The WorkloadMonitor CRD now requires at least one selector label. An
empty selector would match every Pod, PVC, Service and DataVolume in the
namespace. No chart renders one.

When an application has several monitors that are not operational and
only some of them name a cause, `WorkloadsReady` carries the named
messages and reason only. A vm-disk has one monitor, so it does not hit
this.

### Which phases count as ready

Settled phases pass: `Succeeded`, `PendingPopulation`,
`WaitForFirstConsumer`, `Paused`. A disk with `bindImmediately: false`
on a `WaitForFirstConsumer` class rests in `PendingPopulation` or
`WaitForFirstConsumer` until a VM consumes it. A multi-stage import
rests in `Paused` until its next checkpoint, and the vm-disk chart
renders no checkpoint source. These report ready while holding no data,
because that is where such a disk is supposed to sit.

`UploadReady` is not one of them. CDI enters it when the upload server
pod is running and ready, and leaves it for `Succeeded` only when that
pod succeeds (`updateStatusPhase` in
`pkg/controller/datavolume/upload-controller.go` at v1.66.1), so the one
phase covers a disk nobody has uploaded to and a half-uploaded one. A
`source: upload` disk therefore reports `WorkloadsReady=False` from
creation until `virtctl image-upload` completes.

A phase the controller does not know also reads as ready, so a CDI
release that adds or renames a phase cannot turn working disks red. The
cost goes the other way: a renamed in-flight phase silently reads as
settled. The list wants a re-read whenever the CDI version moves. It
matches the named phases of the CDI v1.66.1 API, the version vendored in
`packages/system/kubevirt-cdi-operator`.

### Clusters without CDI

CDI comes with the IaaS bundle, while cozystack-controller runs
everywhere, and CDI can be installed while the controller is running. A
watch on a kind whose CRD is absent never syncs, and then the whole
controller does not start. So the watch is not set up at startup. A
runnable adds the DataVolume watch once the kind is served. It waits for
the first reconcile, because only from then on does a new watch start at
once, and then checks discovery right away and every 30s after that. It
installs the DataVolume reader only after the DataVolume informer has
synced within a timeout, because a reconcile context has no deadline and
a List on an informer that never syncs would block every WorkloadMonitor
reconcile. The watch source is started once: a controller-runtime `Kind`
source adds an event handler to the informer and never removes it, so on
a timeout the next check waits on the source it already has instead of
starting another one. RBAC needs no change: the controller's ClusterRole
already has read on everything.

When the DataVolumes cannot be read, because the reader is not installed
yet or the List fails, the monitor still publishes its replica counts
and keeps its last DataVolume verdict, and a List error is returned so
the request is retried and recorded as a `DataVolumesUnavailable`
Warning event on the monitor. The status of a monitor without
DataVolumes is not affected by a failing List. A disk whose verdict is
already recorded does not read as ready in the window after a controller
restart. A monitor with no recorded verdict yet, which is every monitor
right after the upgrade that brings this change and any disk created
right after a restart, reads ready until the DataVolume watch has
synced. Because the watch starts on the first reconcile rather than on
the next discovery tick, that window is the time to sync the DataVolume
informer plus at most one 5s requeue, not a 30s poll interval. The
window is not zero, since a monitor reconciled in it is published before
its DataVolumes can be read. From then on, a reconcile that runs before
the watch has synced is requeued after a few seconds, so the DataVolume
events that queue it are not lost. A pass that changes nothing in the
monitor status does not write it, so that requeue sends no status writes
to the API server. Once the reader is installed, every monitor that
holds a DataVolume verdict is queued once more. So a verdict whose
DataVolume was deleted while the controller was down is recomputed then,
not at the periodic resync.

### Tests

The table test in `workloadmonitor_controller_test.go` covers the named
phases of CDI v1.66.1, plus Failed, Unknown, an empty phase and a
missing status. A second table checks the message and reason for an
importing, a failed, a phaseless and a populated disk, and a third test
lists two stuck DataVolumes in both orders and expects one message. It
also covers a DataVolume outside the selector, the kind not being
served, a List error that still publishes the replica counts, and a
missing reader that keeps the last verdict. The watch tests check that
the watch starts on the first reconcile and not before it. A source that
never syncs leaves no reader and is retried on the same source, with the
reconciles in between still requeued. A List failure records the Warning
event, the sync window records none, and an unchanged pass writes no
status. A manager that stops during the wait gets the cancellation back,
and a reconcile that runs while the watch syncs is requeued without an
error. A table test covers the PVC Workload: importing, populated, a PVC
with no DataVolume controller or a controller of another kind or group,
and the unread cases. A helm-unittest in
`packages/system/cozystack-controller` pins the selector's
`minProperties`. `rest_table_test.go` pins the `WORKLOADS` column,
`rest_workloads_test.go` the condition message and reason, and
`status.test.ts` the console badge, including a halted VM that stays
green.

In `hack/e2e-chainsaw/vminstance` the existing VMDisk scenarios now wait
for the DataVolume to be Ready, then for `Ready` and `WorkloadsReady` on
the VMDisk itself. That replaces the wait on the HelmRelease. Those
asserts would pass even if the monitor ignored DataVolumes, so a new
scenario imports from an `.invalid` URL. It first waits for the importer
to have failed, which CDI reports on the DataVolume as `Running=False`
with reason `Error`, and then for `Ready=True` with
`WorkloadsReady=False` and reason `DataVolumeNotReady` on the VMDisk.
Without the first wait the VMDisk assert passed in about a second, on a
DataVolume that had no phase yet. The e2e runs on a fresh install only,
so nothing here exercises an existing release across an upgrade.

### What it does not do

#2908 lists possible fix directions, and none of them is here. The
cross-class clone still never completes, it just stops being invisible.
Keep the issue open.

While the DataVolume watch syncs, every monitor is still requeued. Only
the ones whose selector reaches a DataVolume need it, but the controller
cannot tell which those are before it can read DataVolumes.

Closes #4449

### Screenshots

There is no screenshot: I had no running console to take one from. The
badge change is pinned by the vitest cases in
`packages/system/dashboard/images/console/apps/console/src/lib/status.test.ts`:
a VMDisk whose `WorkloadsReady` has reason `DataVolumeNotReady` goes
amber, and a halted VMInstance with the generic reason stays green.

### Downstream repositories

I walked the trigger map against the diff.
cozystack/terraform-provider-cozystack is affected: its `wait_for_ready`
reads only `Ready`, so on a `cozystack_vmdisk` it returns before the
disk is populated once this lands. That needs a decision on the provider
side, so it is an issue there. The release-time bot regenerates
`packages/apps/vm-disk/README.md` into the website's virtualization
pages, and `vm-disk` is in the `VMS` list that generator reads, so the
website needs no follow-up. The ApplicationDefinition keeps its kind,
plural, prefix and path.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/59
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(vm-disk): report whether CDI finished populating a disk in the WorkloadsReady condition, which names the stuck DataVolume and its phase. kubectl get shows WorkloadsReady in a new WORKLOADS column for every application kind, and the console badge turns amber while a disk is not populated, so a disk whose import or clone never completes no longer looks healthy. Ready of a new or updated vm-disk no longer waits for the DataVolume, so wait on WorkloadsReady to know the disk is populated.
```
2026-10-02 19:47:02 +03:00
IvanHunters 339be8d42d fix(vm-default-images,kubernetes-worker-image,vm-disk): wait for golden images to import and refuse cloning a failed one (#4392)
## What this PR does

Both golden image catalogs, `vm-default-images` and
`kubernetes-worker-image`, went Ready before any image had imported.
Each one renders a DataVolume per image with immediate bind, so every
import starts at install, but neither PackageSource had
`healthCheckExprs`. The HelmRelease went Ready as soon as helm applied
the DataVolumes, and an image that failed or stalled stayed green.

This PR makes both releases wait for the imports. Each component's
`install:` block now has a DataVolume health check. `current` holds only
on `Succeeded`, and `failed` holds on `Failed`. Any other phase,
`Unknown` included, matches neither, and flux keeps waiting. A
DataVolume with no `status` key makes the CEL evaluation error out. Flux
reports that as Unknown and keeps waiting too.

The other settled phases are safe to treat as not ready for these
charts. `PendingPopulation` and `WaitForFirstConsumer` are where a
volume waits for a consumer, and immediate bind removes that wait.
`UploadReady` and `Paused` belong to upload and multi-stage sources, and
both charts render only `source.http`.

There is no per-component timeout, so I did not set one. The window is
the operator-wide `--helmrelease-install-timeout`, 10m by default. The
operator sets `RetryOnFailure` on both install and upgrade, so a
download that outlasts the window is retried as an upgrade until the
import finishes. Nothing uninstalls the DataVolumes in between, and the
import keeps running.

One image that never imports keeps its whole catalog not Ready until the
image imports or its entry is removed. There is no time limit, and I
kept it that way on purpose. A group/kind health check cannot tell one
image from another, so it cannot be made per image. A per-component
timeout would be an API change for one opt-in package. Both charts'
`values.yaml` document this. Nothing on `main` lists either package in
`dependsOn`, and the images that did import stay usable in the meantime.
Tenant worker pools look up each golden directly and never wait on the
catalog's status.

The price is a helm-controller slot. The operator labels the generated
HelmRelease with `cozystack.io/package`, plus `cozystack.io/privileged`
when privileged. The flux-shard-operator acts only on HelmReleases that
already carry `sharding.fluxcd.io/key`. So both catalogs go to the
system helm-controller, which picks up releases without that key and
runs with `--concurrent=5`. A DataVolume in `Failed` ends each attempt
at once and costs next to nothing. An import that stalls without
reaching `Failed` is the expensive case, for example an importer that
keeps retrying an unreachable URL. Each attempt waits out the full
window, and the next one starts 30s later. So one of those five slots
stays busy about 95% of the time for as long as the image is stuck.
#4145 describes the same cost on the shard controllers.

The status change alone does not protect a vm-disk, because vm-disk
clones from a golden without checking it. A golden still importing is
fine, since CDI holds the clone until the source is populated. A golden
whose import has Failed is never populated, so the clone would wait
forever. vm-disk now refuses to render a new disk on a Failed golden and
names the image. This matches the check the worker pools already have. A
disk that already exists is exempt, so a golden that fails later does
not break that disk's upgrades. The check fires only when CDI sets
Failed, and at CDI v1.66.1 an import reaches Failed only on a checksum
mismatch. The goldens carry no checksum, so a broken URL keeps the
importer retrying and the new disk still waits, with only the catalog's
not-Ready status pointing at the image. #4564 tracks that gap. A golden
that does not exist at all is a separate gap, tracked in #4548.

Tests pin the expressions for both catalogs. A helm-unittest case checks
them on the rendered PackageSource. A Go test builds the HelmRelease
spec from the shipped file and checks that they arrive with the poller
wait strategy. The repo has no harness that evaluates flux CEL, so the
phase behaviour above is argued from flux's evaluator and not exercised
by a test. The vm-disk check has helm-unittest cases for a Failed, a
Succeeded, and a still-importing golden, for a render with no cluster,
and for an existing disk.

Closes #4169

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(vm-default-images, kubernetes-worker-image): the golden image catalogs now report Ready only after their images have finished importing, and a failed or stalled import keeps the catalog not Ready. A vm-disk created from a golden image whose DataVolume is in phase Failed is now refused with an error naming the image, instead of waiting forever on a clone that cannot complete.
```
2026-10-02 15:44:28 +03:00
Mattia Eleuteri cc7f98142a fix(backups): read the keepOriginalPVC rename back past the cache
renamePVC created the -orig PVC, then read it back through the cached
client to learn its UID. The informer had often not seen the PVC yet, so
the read failed with NotFound. On a two-disk VM restore one disk was
kept and the other was not; with that error turned into an event, its
DataVolume was deleted, the original PVC stayed in place and Velero
skipped it, and the restore reported success without restoring the
disk. The -orig PVC was left Pending on the same PV, by then Retain.

Create already returns the UID, so the read-back goes. The other reads
the rename decides on, the -orig and original PVCs and the PV, go
through the manager's uncached reader, and the PV rebind retries on
conflict, since the PV controller writes the PV once the original PVC
is deleted.

A rename that fails for any reason other than an -orig name taken by
another PVC now requeues the restore instead of failing it. The
DataVolume is still there, and the retry finds its own -orig PVC by the
RestoreJob UID and resumes where it stopped.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-02 14:34:08 +02:00
Aleksei Sviridkin ace69cb075 docs(controller): say what a nil DataVolume reader means
The field comment still said DataVolumes do not count while the reader
is nil, and that the reader appears once the kind is served. Neither
holds: the reader is installed only after the informer has synced, and
until then a monitor keeps its stored verdict, which still decides
Operational.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:38 +03:00
Aleksei Sviridkin a6a32d79cd fix(controller): recompute stored DataVolume verdicts once the reader is in
A reconcile that runs before the DataVolume watch starts finds no
reader, keeps the monitor's stored verdict and is not requeued. The
watch's initial replay then delivers only the DataVolumes that still
exist, so a monitor whose DataVolume was deleted while the controller
was down kept reporting it not ready until the periodic resync, hours
later.

Installing the reader now queues every monitor that holds a DataVolume
verdict, through the controller's own queue, so each is recomputed
against the synced cache. Monitors without a verdict are left alone:
the replay already reaches every DataVolume they could select. If the
monitors cannot be listed, the next discovery poll retries.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:37 +03:00
Aleksei Sviridkin 694f4f2ff9 fix(controller): report a disk's PVC Workload by its DataVolume
CDI copies the DataVolume labels onto the PVC it creates, so the vm-disk
monitor also publishes a Workload for that PVC, and its readiness was
bind state alone. Mid-import the PVC is already Bound: the Workload read
operational while the monitor, and the Application's WorkloadsReady,
said the disk was not populated.

A PVC whose controller is a DataVolume the monitor finds not ready now
reads not operational. While the DataVolumes cannot be read, its
Workload keeps the verdict it has, as the monitor does.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:36 +03:00
Aleksei Sviridkin 366b5e80f4 perf(controller): skip the monitor status write when nothing changed
While the DataVolume watch syncs, every WorkloadMonitor of every kind
is requeued every few seconds, and an informer that cannot sync keeps
that window open. Each pass wrote status unconditionally, so a cluster
with many monitors sent the API server a steady stream of writes that
changed nothing. Comparing against the stored status first leaves the
requeue as cache reads only.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:36 +03:00
Aleksei Sviridkin dd72583dfd fix(controller): record an event when DataVolumes cannot be read
A failed DataVolume List keeps the monitor's last verdict and retries,
and the only trace of it was a controller log line. A monitor with no
verdict yet then reads operational while its disks are unreadable, with
nothing a tenant can see saying why. A Warning event on the monitor
makes the failure visible where the bucket metrics failure already is.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:36 +03:00
Aleksei Sviridkin beeb7994cc fix(controller): start the DataVolume watch source only once
A controller-runtime Kind source adds an event handler to the informer
when it starts and never removes it; cancelling it on a sync timeout
only ends its context. Building a new source on every failed sync
attempt therefore left one more handler on the DataVolume informer per
retry, each enqueueing the same events, until the controller restarted.

The source is now started once and later attempts only wait for the
informer to sync. Because the started source keeps delivering events
between attempts, the syncing flag stays set until the reader is
installed, so a reconcile queued in that gap is requeued as well.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:35 +03:00
Aleksei Sviridkin a9b276f99b fix(controller): requeue monitors reconciled while the DataVolume watch syncs
Starting the DataVolume source replays every DataVolume to its
handler at once, and the handler queues the matching monitors, while
the reader is installed only after the source has synced. A monitor
reconciled in that window saw no reader, kept its last verdict and
returned with nothing left to queue it again, so a disk that failed
or finished while the controller was down could keep a wrong
WorkloadsReady until the next unrelated event.

Such a reconcile still publishes the replica counts and now asks to
be requeued after a few seconds, until the reader is in place or the
sync attempt gives up. It is a requeue rather than an error because
the window is expected: an error would count as a reconcile failure
and log a line for every monitor in the cluster while it lasts.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:35 +03:00
Aleksei Sviridkin 99f3d3b2b7 feat(controller): tag a DataVolume cause with a condition reason
WorkloadsReady=False covers different situations: a stopped VM has
no pod and its monitor reports it not operational on purpose, while a
disk whose DataVolume never populates is broken. A consumer that wants
to react to one and not the other had only the message text to go by.

The WorkloadMonitor status gains a Reason next to Message, set to
DataVolumeNotReady by the DataVolume check, and the Application API
carries it into the WorkloadsReady condition in place of the generic
reason whenever a monitor that is not operational names one.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:33 +03:00
Aleksei Sviridkin 81ffcda86f fix(controller): keep publishing monitor status when DataVolumes are unread
The DataVolume List sits on the reconcile path of every
WorkloadMonitor, and returning its error before the status write froze
the replica counts and Operational of a monitor that selects no
DataVolume at all whenever CDI's API hiccupped.

The status is now written either way, with the last DataVolume verdict
kept whenever the DataVolumes could not be read, and the error still
returned so the request is retried. The same applies while no reader
is installed yet, so a stuck disk does not read as ready for the
window between a controller restart and the DataVolume informer
syncing.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:32 +03:00
Aleksei Sviridkin 48ece79ffc feat(controller): name the DataVolume that keeps a disk not ready
For a vm-disk, WorkloadsReady is the one condition that reports
whether CDI populated the disk, and its generic "One or more workloads
are not operational" gives an operator nothing to act on: a disk has
no workloads, and the stuck DataVolume and its phase are exactly what
they need to know.

The WorkloadMonitor status gains a message that the controller fills
with each selected DataVolume that is not ready and its phase, and the
Application API uses the messages of the monitors that are not
operational in WorkloadsReady instead of the generic text whenever one
of them names a cause.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:32 +03:00