Commit Graph
7331 Commits
Author SHA1 Message Date
Myasnikov Daniil 9fe93a3d7e docs(operator): fix three claims that overstated what the code does
The static-pod note said containers built from an on-disk manifest are
unaffected "in both directions". True of the container, which the
kubelet runs from disk whatever the API says, but the mirror pod is an
ordinary API object that passes admission like any other, so a static
pod declaring a memory request above the ceiling with no limit of its
own has its mirror pod rejected -- leaving the container running with no
representation in the API. Worth naming, since kube-system is in scope
by design. The scan does cover it: a mirror pod is a live Pod.

The ReplicaSet exclusion was justified on live ones being copies of
their Deployment's template. That does not reach an ownerless
ReplicaSet, which at replicas 0 has neither pods to be seen through nor
a Deployment template standing in for it. It stays out of scope --
nothing here ships a bare ReplicaSet -- but as a gap left open rather
than one the stated reason closes.

The delete guard's comment claimed the enabled apply always adopts and
stamps a same-named foreign object. It does not when that object's spec
already matches, because then the apply is skipped entirely. The
outcome is the same one the guard gives on purpose, but it is reached by
the apply not running.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:54 +03:00
Myasnikov Daniil fe792369e5 docs(platform): bring the memory-limit page in line with the operator
docs/operations/system-memory-limits.md merged ahead of the code it
describes, so main documents two installer knobs and a 4Gi default that
exist nowhere outside this branch. An operator following the shipped page
today sets a value that silently does nothing.

The page is corrected to the default this branch ships and gains the
three behaviours it had no account of at all: that a namespace holding a
request above the ceiling is raised rather than skipped, that a raised
ceiling only comes back down after the scan has been continuously empty
for a grace period and why that delay has to exist, and that a namespace
another LimitRange already governs is left alone. Each names the log line
to grep for, since none of the three is visible in the object.

The ceiling rationale is also stated more honestly: the scan reads
declared requests and never a container's real resident set, which is the
reason the default sits far above any plausible working set rather than
close to one.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:54 +03:00
Myasnikov Daniil 1df6e3ac8f revert(linstor): leave the controller memory limit to its own PR
This branch gave linstor-controller a 10Gi memory limit, justified in
the values and pinned in cluster_test.yaml on the grounds that the
namespace default was 4Gi, half the -Xmx8G baked into the image.

Both halves of that have moved. The default is now 32Gi, four times the
heap ceiling rather than half of it, so nothing about the JVM requires a
per-chart limit to sit above the namespace default any more. And #3824
argues the opposite direction for the same component -- extending the
controller's existing limit-free exemption to linstor-satellite, using
that exemption as its precedent. Keeping the 10Gi here would remove the
precedent that PR reasons from, and whichever of the two merged second
would silently invalidate the other's rationale.

So linstor policy belongs to one PR, and it is not this one. The three
files go back to their state on main; this branch is now confined to
the operator and the installer chart.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:53 +03:00
Myasnikov Daniil e131a49514 feat(operator): default the system namespace memory ceiling to 32Gi
The guard in front of this default reads declared memory requests. It
cannot see a container's real working set, so a system component that
requests little but legitimately uses more than the ceiling, and
declares no limit of its own, was handed a deterministic kernel OOM on
its next recreation in exchange for leaving the Talos handler's victim
set. Since the feature ships enabled, that trade fell on the whole
fleet on upgrade.

It is not a trade this feature has to make. The handler discards a
cgroup for carrying a memory.max at all, whatever its value, so
immunity does not require a fitted ceiling -- only a real one. A
default far above any plausible working set buys the immunity and
leaves nothing for the ceiling to bind on, which also keeps it clear of
the failure mode this repository already carries a scar from, where an
undersized limit converted a rare pressure-driven kill into a reliable
one.

Nothing about scheduling changes: only requests are scheduled against,
and the paired default request stays 32Mi. An operator who wants the
ceiling to bind lowers it, per component in a chart or fleet-wide with
this knob, once real usage is measured -- which is the direction that
needs evidence, and now the direction that needs it rather than the
default.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:53 +03:00
Myasnikov Daniil ad3c9ef04e fix(operator): stay out of a namespace another LimitRange governs
The reconcile only read its own LimitRange by name, so a system
namespace already carrying an administrator's LimitRange got a second
one beside it. LimitRanger applies defaults from every LimitRange in
the namespace and then validates the pod against every one, and nothing
orders them, so the effective container ceiling moved between the
administrator's value and this one for reasons no chart author could
see. That is the same hazard this feature cites to keep itself out of
tenant namespaces, and it does not stop being one because the namespace
is a system namespace.

A max is the sharper half. This reconciler writes a default and no max,
so an administrator's LimitRange carrying a container memory max below
the configured default had every container that declares no limit of
its own rejected: the default is written first, then validated against
the max it exceeds. That takes out a whole namespace, and it reaches
kube-system, which this feature enters for cozystack-scheduler and
which is where a distribution's or an administrator's own LimitRange is
likeliest to sit.

Such a namespace is now left alone and logged, and anything this
reconciler wrote there earlier is withdrawn rather than left sitting
beside the foreign object, since that object is half of the ambiguity.
The check is deliberately narrow -- only a container memory default or
max collides with what is written here -- because read more widely it
would be a blanket opt-out that an unrelated cpu policy could trip.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:52 +03:00
Myasnikov Daniil 5630954e34 fix(operator): hold a raised memory ceiling before lowering it
A raised container memory ceiling was recomputed from the scan on every
reconcile, so a single scan that found nothing lowered it back to the
configured limit. An empty scan is not evidence that the oversized
request is gone: it is equally consistent with the pod being between
deletion and recreation.

For a blocker that appears only as a live Pod, which is a class this
repo ships -- every CNPG Cluster in a system namespace builds its
instance pods directly rather than through a StatefulSet -- an ordinary
Talos node drain opens that window, and reconciles follow HelmRelease
status churn, of which a node upgrade produces plenty. The ceiling was
lowered under a pod that was merely being rescheduled; its recreation
was then defaulted a limit below its own request and rejected at
admission for good, because the rejected pod never exists so no later
scan can find it, and the namespace settles at the configured limit
where the pod half of the scan stops running at all. The drop was not
even logged.

Lowering now waits until the same emptiness has held for
memoryCeilingDropGrace, with the instant it was first observed recorded
on the object so it survives an operator restart and a change of
leader. A wall clock rather than a count of consecutive clean scans,
because the reconcile rate rises exactly when the risk does and any
small count can be burned inside one rescheduling window. A blocker
seen again cancels the countdown, an unreadable stamp restarts it, and
both the hold and the drop are logged.

The steady-state comparison now also covers the two metadata fields
this reconciler manages, because the grace stamp is one of them. That
incidentally fixes the managed-by label never being re-stamped when
only metadata had drifted, which had left turning the knob off unable
to remove an object the operator did own.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:52 +03:00
Myasnikov Daniil b6cb0def78 docs(operator): say that kube-system is in scope for the memory default
The scope of the default LimitRange is every namespace a Package targets
whose name does not start with `tenant-`, and cozystack-scheduler installs
into kube-system, so a namespace owned by the underlying Kubernetes
distribution is defaulted like any other. Neither the flag help nor the
installer value said so, and both are what somebody setting the knob
reads.

docs/operations/system-memory-limits.md already covers this, including the
part that surprises: containers the kubelet builds from an on-disk
manifest never pass through admission, so a Talos control-plane static pod
gets no memory.max from this feature in the first place and the mirror pod
is the only thing the LimitRange reaches. Both notes now carry the same
statement, so neither reading of the knob leaves it out.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:51 +03:00
Myasnikov Daniil 3c1d8d6988 docs(operator): bound the managed-by guard to the disabled state
The comment on the label check in deleteSystemDefaultsLimitRange reads as
a general promise not to touch an administrator's own LimitRange of that
name. It is narrower than that, and the difference is worth writing down:
while the feature is enabled the apply takes the name over with
ForceOwnership and stamps the managed-by label itself, so a hand-written
cozystack-system-defaults is adopted and then deleted here when the knob
goes off.

That is the intended SSA behaviour, not a defect. What the label actually
buys is that a LimitRange of that name the reconciler never wrote to
survives the knob, which is a real guarantee and a smaller one.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:51 +03:00
Myasnikov Daniil accb4f9e1b refactor(operator): drop the dead error check from the settled-state guard
The guard read `err == nil && apiequality.Semantic.DeepEqual(...)`, which
looks like it consults the earlier Get and does not. That Get's error was
reassigned by the scan two statements up, and the scan's non-nil case has
already returned, so the clause is always true by the time it is
evaluated.

It was harmless only by accident: a namespace whose Get returned NotFound
carries an empty spec, which never compares equal to the desired one, so
the apply happened anyway. Removing the clause makes that the reason
rather than a coincidence sitting behind a condition that cannot fire.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:50 +03:00
Myasnikov Daniil ead330ccc3 fix(operator): keep scanning pods while the ceiling stays raised
The pod half of the memory scan was skipped whenever any container memory
default was in force. That is the weaker of the two available conditions,
and it silently retracted a ceiling this reconciler had just raised.

The default written into a system namespace is either the configured limit
or a ceiling raised above it to clear an oversized request, and the raise
is recomputed from the scan every reconcile rather than read back off the
object. So a raise that only the pod half could justify -- a live pod
requesting more than the limit with no limit of its own, in no workload
template the scan can read, which is what an operator building pods from
its own custom resource produces -- lasted exactly one reconcile. The
raised LimitRange satisfied "in force", the next reconcile skipped the pod
list, the scan came back empty, and the configured limit was reapplied
underneath a still-running pod. That pod's next recreation was rejected
with "must be less than or equal to memory limit", and because the skip
stayed in effect nothing ever raised the ceiling again: a healthy
component goes down at a node reboot and stays down, with the revert
unlogged.

Lowering the knob reached the same place with no race in it. An operator
running at 16Gi has pods admitted with requests only that ceiling allowed;
dropping the flag to 4Gi wrote 4Gi straight over them.

Gate the skip on the namespace already defaulting memory at exactly the
configured limit instead. That keeps the pod list alive for as long as a
namespace sits raised, which is what lets the raise be re-justified or
dropped on evidence rather than frozen or forgotten, and it still skips
the list in the steady state the skip was added for. The comparison is by
value for the same reason the spec comparison is semantic: 4096Mi and 4Gi
are one ceiling.

The cost is that a raised namespace reads the pod list every reconcile.
That is the price of being able to lower the ceiling again, bounded by how
many namespaces sit raised -- none, once the oversized requests are gone.

Three tests cover it, each of which fails on the old condition: a raised
ceiling held while the pod needing it runs, the same ceiling dropping back
once that pod is gone, and the configured limit falling below a ceiling
already in place. A fourth pins the notation equality. The fixture comment
on the steady-state test claimed its 8Gi pod could not exist beside a
settled 4Gi LimitRange; it can, through a narrow window, so it now says
so.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:50 +03:00
Myasnikov Daniil 26a1ce5be0 test(operator): drop the incidental fixture from the no-pods case
The scaled-to-zero test seeded a spec-less LimitRange, which is not part
of the case it names. It is the first application in the namespace that
makes a workload with no pods invisible to a pod-only scan, so start from
no LimitRange at all and let the assertion rest on the workload rather
than on the fixture.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:50 +03:00
Myasnikov Daniil b62fac474c refactor(operator): move both server-side applies off the deprecated client.Apply
client.Apply as a patch type is deprecated in controller-runtime v0.23.1
in favour of Client.Apply taking a runtime.ApplyConfiguration. Convert
both call sites — the system defaults LimitRange, new on this branch, and
the Namespace apply that predates it — keeping the same field owner and
force-ownership options.

The LimitRange keeps its typed builder, because that is what the spec
comparison and the tests read, and gains a small function that restates
it as an apply configuration immediately before the wire. The Namespace
apply now sends only the labels and annotations the reconciler actually
sets, rather than a whole typed object whose spec and status were empty.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:49 +03:00
Myasnikov Daniil c7af79b734 fix(operator): only delete the LimitRange this reconciler owns
deleteSystemDefaultsLimitRange matched on name alone, and the name is not
reserved: nothing stops an administrator creating a LimitRange called
cozystack-system-defaults by hand. Turning the knob off would then
silently remove policy this operator never wrote, in every system
namespace at once.

Stamp app.kubernetes.io/managed-by on the object the reconciler applies
and require it before deleting. Anything else is left where it is, which
is also the right answer if the name is ever taken over by another
component. Nothing in the wild carries the old unlabelled shape, since
the feature has not shipped.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:49 +03:00
Myasnikov Daniil dbc5e5ea13 perf(operator): read before deleting the disabled LimitRange
With the limit set to 0 the disabled path runs for every system namespace
on every Package reconcile and issued a DELETE each time, swallowing the
404 that nearly all of them returned. That is a write attempt per
namespace per reconcile, forever, and every one of them lands in the
audit log.

Get from the informer cache first and return when there is nothing there.
The test now counts deletes, so a second reconcile over an already-empty
namespace has to stay at zero rather than merely not erroring.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:48 +03:00
Myasnikov Daniil 89b78cbf20 docs(operator): drop the duplicated field from the ceiling-raise log
request and raisedLimit were logged with the same value, because the
raised ceiling is the blocking request. Keep request, which names the
thing an operator has to go and change, and say in the message that the
ceiling becomes it.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:48 +03:00
Myasnikov Daniil 9866113da0 test(operator): pin the system namespace memory flag parsing
parseQuantity and the request-above-limit check were closures inside
main(), so no branch of either was reachable from a test: not empty, not
malformed, not negative, not a request of 0 against a real limit, and not
the pair check itself. That last one is the one worth having. A limit and
a request that each parse but contradict each other produce a LimitRange
the API server rejects, once per system namespace on every Package
reconcile, and nothing in that error points back at a flag.

Lift both into parseSystemNamespaceMemory, which returns an error instead
of calling os.Exit, and pin all of it: the shipped defaults, empty as the
off switch, an explicit 0 limit beside a set request, a 0 request against
a real limit (supported on purpose), malformed and negative values on
either flag, request above limit, and request equal to limit as the legal
boundary. main() keeps the same exit-on-error behaviour, now in one place.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:47 +03:00
Myasnikov Daniil dd36eeddbc docs(operator): describe the raise, not the retract, in the shipped help
The flag help and the installer values comment both still said the
operator skips a namespace holding a request above the configured limit
and retracts any LimitRange it applied there. It stopped doing that in
3127c0fad: it raises that namespace's ceiling to clear the request and
drops back once the request is gone. An operator reading the shipped help
and then finding an 8Gi LimitRange in cozy-monitoring had no way to
connect the two.

Both texts now describe the raise, the drop-back and the log line to grep
for. The values comment also loses its VPA warning, which this branch's
own comment in findRequestAboveDefaultLimit refutes — VPA rescales the
limit in proportion to the request under the default controlledValues, so
it cannot be what puts a namespace over — and gains a pointer to
declaring a limit in the component's own chart, which is how
packages/system/linstor now stays out of the default's reach.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:47 +03:00
Myasnikov Daniil ae431808ea fix(linstor): give linstor-controller a memory limit above its baked-in -Xmx8G
linstor-server bakes `-Xms256M -Xmx8G -XX:+CrashOnOutOfMemoryError` into
the generated bin/Controller (build.gradle, jvmControllerOpts), nothing in
this chart or in piraeus-entry.sh overrides it, and a JVM handed an
explicit -Xmx never reads memory.max. The container was deliberately left
with a 700Mi request and no limit at all, which was the safe choice while
nothing else supplied one.

It stops being safe alongside the system-namespace LimitRange: a container
declaring no memory limit is given the configured default, 4Gi, so the
process's own heap ceiling would sit at twice its cgroup ceiling and a
large enough DRBD resource count would have the kernel kill the storage
control plane where the JVM would otherwise have crashed itself loudly at
8G. Stating the limit here is also what keeps the container out of the
LimitRange's reach, since a container with its own limit is never
defaulted one.

10Gi is the 8G heap plus room for metaspace, code cache, thread stacks and
direct buffers. CPU stays unlimited. The helm-unittest assertion that
pinned "no limits" is replaced by one pinning the memory limit and the
absence of a CPU limit, so neither lowering it below the heap ceiling nor
dropping it again passes silently.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:47 +03:00
Myasnikov Daniil 87f79c17c3 fix(operator): keep scanning templates after the LimitRange settles
The steady-state fast path returned as soon as the LimitRange matched the
configured spec, on the grounds that LimitRanger gates anything a scan
could still find. That holds for pods and not for templates: a Deployment
added to a settled namespace with a memory request above the ceiling and
no limit of its own is admitted exactly as written, and only the pods it
then tries to create are rejected with "must be less than or equal to
memory limit". Nothing raised the ceiling, because every later reconcile
returned at the same line, so a chart bump became a rollout that could
never produce a pod and logged nothing about it.

Split the scan instead. Templates are read on every reconcile; the pod
list, which is the expensive half, is read only while no memory default
is in force in the namespace — once one is, LimitRanger fills in a limit
for every container admitted without one, so a container carrying a
request with no limit beside it can no longer come into existence. The
skip keys on the default actually being in force rather than on the
object existing, so a LimitRange whose spec was stripped scans pods again.

The write is now guarded by a semantic comparison against the spec the
scan settled on rather than against the configured spec, so a namespace
already sitting at a raised ceiling stops taking an identical apply on
every reconcile as well.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:46 +03:00
Myasnikov Daniil c9cacda1a0 docs(operator): pin the admission order by rule, not plugin index
The scan's justification for treating a VerticalPodAutoscaler as harmless
rested on LimitRanger being "position 6 of AllOrderedPlugins" and
MutatingAdmissionWebhook being "35". Both numbers are properties of one
release rather than of the ordering, and they had already drifted out
from under the comment. Under k8s 1.35, which this tree builds against,
LimitRanger is the 6th entry but MutatingAdmissionWebhook is the 33rd;
index 35 is ResourceQuota and the 35th entry is ValidatingAdmissionWebhook,
neither of which mutates, so the sentence named the wrong plugin under
either counting convention. In 1.31 the same plugin sat at 30, and
SecurityContextDeny being dropped in 1.35 is what moved LimitRanger.

Restate it on the invariant the list asserts about itself -- webhook,
resourcequota and deny plugins must go at the end -- with LimitRanger
compiled in above that marker and every plugin that can rewrite a
container's resources below it. That survives a release bump where an
index does not, and it covers MutatingAdmissionPolicy as well, a second
mutator that the positional wording predated and never accounted for.

Record what the order buys, which the old wording left implied. Run the
other way round, a plugin mutating before LimitRanger would write a
request into a container that has no limit yet, LimitRanger would then
default the limit to the configured ceiling, and core validation would
reject the fully mutated pod. The scan can see none of that -- the
request is in no template and the pod is never admitted -- so the order
is the only thing keeping VPA out of the residual gap this comment
already records for controlledValues: RequestsOnly.

Fix the matching rationale on the live-pod case in
TestFindRequestAboveDefaultLimit, which still claimed the pod's request
was raised by a webhook after the template was rendered. The ordering
rules that out; what the case actually rests on is a pod admitted before
the LimitRange existed, or one its controller builds from a custom
resource no template in the scan can show.

No behaviour change, and no new test: a container that carries its own
memory limit is already pinned out of the scan's scope for a template,
in TestFindRequestAboveDefaultLimit, and for a live pod, in
TestReconcileSystemDefaultsLimitRangeAppliesWhenLargeRequestCarriesItsOwnLimit.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:46 +03:00
Myasnikov Daniil 7eecbe9b4c docs(operator): document a zero default memory request
--system-namespace-memory-request=0 passes the startup check, since 0 is
not greater than the limit, and produces a LimitRange with
defaultRequest: 0. Whether that is a footgun to reject or a supported
setting to explain was open, so I asked a real API server: envtest with
kube-apiserver 1.36.0 and LimitRanger enabled, with a second namespace
using 32Mi as the positive control that the plugin was actually running.

It is supported. The API server accepts the LimitRange and stores
defaultRequest as 0 rather than folding it back to the default -- the
Default-to-DefaultRequest copy only fills keys that are absent, and this
one is present. A container declaring no memory of its own is then
admitted with an explicit request of 0 and a limit of 4Gi, QoS Burstable.
So LimitRanger does not read zero as absent, and the container still gets
memory.max on its cgroup, which is the whole reason this feature exists.

What it costs is the scheduling signal: nothing is reserved for that
container, which is exactly where system components stood before this
feature. Say so in the flag help text, where an operator setting the knob
will see it, and note at the startup check that 0 passing is deliberate so
nobody hardens it into a rejection later.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:45 +03:00
Myasnikov Daniil 16b8ea4f0e docs(operator): correct the VPA rationale on the request scan
The comment argued that the danger this scan guards against is a VPA
webhook writing a request above the defaulted limit, which is then
rejected at admission. Admission ordering rules that out. LimitRanger is
position 6 of AllOrderedPlugins and MutatingAdmissionWebhook is 35, so
the LimitRange has already defaulted the container's memory limit before
VPA is called, and VPA rescales that limit in proportion to the request
it writes (GetProportionalLimit, under the default controlledValues:
RequestsAndLimits). Request and limit stay consistent and the pod is
admitted.

Rewrite it around what can actually happen: a request already in the spec
when it reaches admission with no limit beside it. In this tree that means
an operator lowering --system-namespace-memory-limit below a statically
declared request -- the largest is 2Gi, in
packages/system/rabbitmq-operator, so the 4Gi default clears them all
today. Two narrower shapes are real too: a pod that predates the
LimitRange, whose own controller may build it from a custom resource no
workload template in this scan can show; and a VPA set to
controlledValues: RequestsOnly, where nothing rescales the defaulted limit
and the rejection returns. The second doubles as the residual gap, since
a first pod rejected outright appears in neither a template nor a pod
list.

The scan itself is unchanged; only the justification was wrong.

Also record why withholding the LimitRange had to go rather than merely
being suboptimal: in a namespace with no LimitRange nothing defaults a
memory limit, so the next VPA-sized pod arrives with a large request and
no limit, the scan finds it again, and the namespace is withheld again.
The old behaviour could never get its foot in the door in cozy-monitoring.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:45 +03:00
Myasnikov Daniil d682158d1f test(operator): pin the tenant namespace boundary
One strings.HasPrefix keeps this feature out of tenant namespaces, and
nothing tested it: "tenant-" appeared twice in package_reconciler.go and
not once in the test file, so replacing the check with isSystem := true
left the whole package green.

What it protects is not subtle. packages/apps/tenant ships
tenant-range-limits, which defaults container memory to 128Mi in every
tenant namespace that declares resourceQuotas. A system LimitRange
landing beside it leaves two LimitRanges defaulting the same containers,
and which of them wins is decided by the order LimitRanger happens to
iterate them -- neither chart controls that -- so a tenant container's
default memory limit would move between 128Mi and 4Gi with nothing in the
tree changing.

Reconcile a variant with one system and one tenant namespace, and assert
no LimitRange is written for the tenant one. The system namespace in the
same run is the positive control: without it the test would still pass if
LimitRanges stopped being written anywhere at all. The cozystack.io/system
label is asserted too, since the same boundary gates both.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:44 +03:00
Myasnikov Daniil 2e1ab6682c fix(operator): raise the namespace ceiling instead of retracting it
When a container in a system namespace requests more memory than the
configured default, the guard used to withhold the LimitRange and delete
any it had applied earlier. That leaves the namespace with no default
memory limit at all, which is the precise state this feature exists to
remove: without memory.max on the pod cgroup the Talos OOM handler keeps
the pod in its victim set, and a tenant workload's memory pressure gets
paid for by a system component.

Default that namespace's memory to the offending request instead. A
looser ceiling still sets memory.max, so the pod scores zero in victim
selection exactly as it would at the configured limit -- 8Gi protects it
as well as 4Gi does, and both protect it infinitely better than nothing.
The raise needs no knowledge of where the request came from, and it
undoes itself: once the request is gone the scan finds nothing, desired
falls back to the configured limit, and the next reconcile applies it.

This is not a hypothetical namespace. packages/system/monitoring pins
maxAllowed.memory at 8Gi for vmselect and vmstorage,
packages/system/monitoring-agents pins 6G for vmagent, all three run
updateMode: Initial, and no monitoring template declares a memory limit.
Initial never evicts, so those requests do not come down on their own:
under the old behaviour cozy-monitoring, which holds the largest memory
consumers on the platform, was permanently excluded from the feature on
exactly the clusters it was written for.

A raised namespace no longer matches the base desired spec, so it scans
on every reconcile. That is deliberate: the raise has to track the
current largest request in order to be able to drop back, and caching it
would freeze a ceiling that nothing would ever lower again. The set of
raised namespaces is small.

The log moves from Error to Info. Nothing is wrong when this fires; the
operator is reporting a ceiling it chose.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:44 +03:00
Myasnikov Daniil c65db1ad52 perf(operator): skip the memory scan when the LimitRange is settled
reconcileSystemDefaultsLimitRange runs for every system namespace on
every Package reconcile, and Package reconciles follow HelmRelease status
churn through Owns, so on a platform with dozens of system namespaces the
pod and workload scan behind the guard is by far the most expensive thing
in the function -- and it is uncached by design, because caching it would
mean a cluster-wide Pod informer in an operator whose job here is to
bound memory.

Build the desired LimitRange first, read the existing one from the
informer cache, and return without scanning when the two specs already
match. That is safe because the LimitRange being applied is itself the
guarantee: from that moment LimitRanger rejects any new pod whose memory
request exceeds this default, so the only requests a scan could still
find are ones that predate the LimitRange, on pods already running, and
there is nothing it would do differently about them. Steady state becomes
one cached GET.

The comparison is apiequality.Semantic, not reflect.DeepEqual, because
resource.Quantity carries its parsed format and cached string: 4Gi and
4294967296 are the same ceiling and must not read as drift, or every
namespace would scan and re-apply forever.

The test counts List calls rather than only checking the resulting
LimitRange, which cannot tell a skipped scan from a repeated one that
reached the same answer, and seeds an 8Gi pod so a scan that ran would
also change the object.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:44 +03:00
Myasnikov Daniil b55c4f347a docs(operator): keep the residual-gap note on the request scan
The pod-only scan documented what it could not see: a request that only
ever exists post-admission, on a workload whose first pod is rejected
outright. Reading pod templates narrows that gap rather than closing it,
and the rewritten comment dropped the note entirely, which reads as a
claim that nothing is left.

Restore it in its narrower form: a template under the limit, a webhook
that raises the request past it, and a first pod rejected before it
exists, is still invisible to this scan.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:43 +03:00
Myasnikov Daniil 10acfe025d test(operator): cover both terminal pod phases in the skip
findRequestAboveDefaultLimit skips a pod in PodSucceeded or PodFailed,
because neither is ever recreated from that spec. Only the Succeeded half
had a fixture, so deleting the PodFailed clause left the whole ./internal
suite green -- half the condition was uncovered.

Give the case a Failed pod as well. Deleting either clause now fails it.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:43 +03:00
Myasnikov Daniil 654412acd2 fix(operator): scan workload templates for oversized memory requests
reconcileSystemDefaultsLimitRange withholds the LimitRange when a
container in the namespace requests more memory than the configured
default limit, because LimitRanger would then default a limit below that
request and the API server rejects the pod. The scan behind that decision
listed pods only, so it could not see a workload that currently has no
pods: a Deployment or StatefulSet at replicas: 0, a CronJob between runs,
a Job with spec.suspend. Those are ordinary steady states rather than
corner cases. The namespace looked safe, the LimitRange was applied, and
the pod was rejected at the next scale-up or the next schedule with "must
be less than or equal to memory limit of 4Gi" -- a workload that cannot
come back, discovered at the moment somebody needs it.

Scan the pod templates of Deployments, StatefulSets, DaemonSets, CronJobs
and Jobs as well as live pods. Both halves are needed: a template is the
only place a scaled-to-zero workload appears at all, and a live pod is
the only place a request that the VPA admission webhook raised after
admission appears. ReplicaSets are deliberately left out -- a live one
copies its Deployment's template, and the older revisions a Deployment
keeps carry superseded templates that only an explicit rollout undo would
ever instantiate, so scanning them would withhold the LimitRange over a
request replaced releases ago. Jobs are named separately because not
every Job has an owner whose template would cover it, and a finished one
is skipped for the same reason a terminated pod is. A suspended CronJob
is not skipped: suspend is the same dormancy as replicas: 0, undone by
the same kind of human action, and exempting it would leave the gap open
for one kind.

Templates are scanned before pods and the comparison stays strict, so a
workload and its own pods report once, as the workload -- the object an
operator has to edit, and one that outlives pod names. A live pod
outranks its own template only when its request is strictly larger, which
is exactly the webhook-injected case. memoryRequestBlocker therefore
carries Kind/name instead of a bare pod name, and the log key is
"workload" rather than "pod".

Every added List fails closed, which the caller already relies on: a scan
that cannot read is not evidence that applying is safe, so the namespace
is skipped and logged instead. The test for that seeds a sentinel
LimitRange, because asserting the object still exists cannot tell "left
alone" from "applied blind" -- a swallowed List error applies a fresh
LimitRange that exists just the same.

RBAC needs no chart change: the operator's ServiceAccount is bound to
cluster-admin, so it can already list these kinds. The kubebuilder
markers are extended anyway to record what the controller reads.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:42 +03:00
Myasnikov Daniil 7cc090b05f test(operator): pin the largest-offender scan to all three fields
The "largest offender is reported" case asserted only the container
name, and systemPod named every container "app", so the assertion held
whichever of the two pods findRequestAboveDefaultLimit had picked. The
comparison that keeps the largest request was uncovered.

Name the fixture container after its pod, assert the reported pod,
container and quantity together, and give the case three offenders whose
largest sits in the middle of the name-ordered pod listing, so keeping
the first, the last or the smallest offender all name a different pod
and quantity. The init-container case now pins the same three fields.

No production behaviour changes.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:42 +03:00
Myasnikov Daniil 546d8e6a54 feat(operator): default system namespace memory via LimitRange
The Talos userspace OOM handler (v1.12+) picks its victim by ranking
cgroups with `memory_max.hasValue() ? 0.0 : {Besteffort: 1.0, Burstable:
0.5, ...}[class] * memory_current`, discarding every cgroup that scores
zero. Only a memory limit sets memory.max on the pod cgroup, so only a
limit takes a pod out of the candidate set: a request merely demotes
BestEffort to Burstable, which under the v1.13.6 default
strictCgroupClassOrdering still means "killed second". PriorityClass has
no effect at all, because the handler walks cgroupfs and never talks to
the API server.

Victim selection is decoupled from the trigger, so the cgroup that
caused the pressure and the cgroup that gets SIGKILLed are unrelated by
design. Tenant workloads generally carry limits and system components
generally did not, which inverted the intended order: a tenant workload
thrashing against its own limit drives node-wide memory PSI, and
metallb-speaker or linstor-satellite is killed for it while the workload
responsible is never touched.

An audit found roughly 70 workloads across ~30 packages with at least
one container missing a memory limit, so per-chart resources would
neither cover the current set nor keep covering it. Maintain a
LimitRange in every system namespace instead, at the single choke point
that already creates them and already distinguishes system from tenant
(reconcileNamespaces). Tenant namespaces are left alone because
packages/apps/tenant owns LimitRange policy there.

The limit is a ceiling, not a reservation: it defaults well above real
usage (4Gi against a largest observed system request of 512Mi).
defaultRequest is set explicitly and small, because Kubernetes otherwise
defaults each request to the limit and reserves that much at schedule
time. Setting the limit to 0 disables the LimitRange and removes any the
operator previously created.

Two things keep the feature from being able to hurt more than it helps.

A LimitRange default only reaches a container that declares no memory
limit of its own, and the API server then rejects a request above its
limit. So before applying, scan the namespace for a container that
requests more than the configured limit and sets none, and where one
exists skip the namespace, retract any LimitRange applied earlier, and
log the pod, container, request and limit. This is not hypothetical:
packages/system/monitoring and packages/system/monitoring-agents ship
VPAs for cozy-monitoring with maxAllowed.memory of 8Gi and 6G under
updateMode: Initial, so the VPA admission webhook -- which runs after
the in-tree LimitRanger -- can write a request past 4Gi that no chart in
the tree declares. That is why the check runs against live pods rather
than rendered templates, and why it reads through APIReader rather than
starting a cluster-wide Pod informer.

And no failure inside the feature propagates: a LimitRange error, or a
pod list that cannot be read, is logged and stepped over instead of
failing the Package reconcile. A namespace without a LimitRange is back
to the behaviour of every earlier release, while a namespace that never
reconciles leaves its components uninstalled.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:41 +03:00
Aleksei Sviridkin d2e3eb4f82 fix(kubernetes, e2e): unblock autoscaling below min size, size the computeplane pool, fix the backupcontroller build (#4768)
## What this PR does

Makes the backupcontroller test package compile again on main. #4587
added a flavor argument to `createCNPGBackupArtifact` and
`marshalCNPGBackupSnapshot`, and #4669, whose CI ran before that, added
two test calls with the old argument lists. Both PRs were green on their
own, and together they break `go test ./internal/backupcontroller/`. The
two calls now pass the same default flavor as the other tests in the
file.

The second commit fixes an e2e suite that fails the required E2E check
here. It is the `computeplane` suite from #4713, and its worker pool was
one node with a single vCPU. After the kubelet reservations that node
has 900m of CPU to allocate. The tenant addons (cilium, coredns, csi,
ingress-nginx, metrics-server, cert-manager) request exactly 900m
between them. If cert-manager's startupapicheck hook pod is created
last, it can't be scheduled and the install times out after 10m. The
reinstall after remediation runs into the same full node. Whether that
pod comes last depends on install order, so the suite passed on #4713
and failed here.

The pool now has two workers with one vCPU each, the same shape the
kubernetes suites use. One vCPU per worker stays on purpose, because of
#3513.

Two workers exposed a deadlock in the tenant cluster-autoscaler. The
suite's pool is created at zero replicas and then gets a min size of 2,
which CAPI does not apply to the live replica count on upgrade (#4786).
The autoscaler's scale-up for a pending pod asks for one node, and the
clusterapi provider refuses it because one is under the min size,
although it is an increase. That refusal backs the group off, and the
min-size enforcement skips groups in backoff, so the pool never leaves
zero. Upstream had the same report in kubernetes/autoscaler#4942, closed
as stale, and it is now tracked in kubernetes/autoscaler#10400. Our
autoscaler image already patches the mirror case for max size, and the
patch now refuses only a decrease below the min size.

`go vet` and `go test ./internal/backupcontroller/...` pass, and so do
the bats pins for the computeplane suite. The autoscaler patch adds a
test for each direction, and the clusterapi package tests pass with it
applied to 1.32.0.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved cluster autoscaling for worker groups outside configured size
limits: groups below their minimum can scale up toward it, and nodes can
be removed when a group exceeds its maximum. Scale-down requests below
the minimum remain restricted.
* **Tests**
* Updated backup and restore test calls with PostgreSQL and empty flavor
values.
* Updated ComputePlane checks for a two-worker pool and clarified
cert-manager installation timing expectations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 04:36:10 +03:00
Aleksei Sviridkin 82ae9f6cf7 fix(kubernetes): let the autoscaler grow a node group that sits below min
The clusterapi provider refuses any SetSize under the group's min size,
including an increase. A pool whose min size is raised while it stands
at fewer replicas then deadlocks: the scale-up for pending pods asks for
fewer nodes than the min, is refused, and puts the group in backoff,
and the min-size enforcement skips groups in backoff. The group stays
where it is for good.

Refuse only a decrease below min size. The guard it keeps is the one
DecreaseTargetSize relies on, and DeleteNodes checks min size itself.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-07 00:58:11 +03:00
Aleksei Sviridkin 4c3c838a66 test(e2e): run the computeplane pool at two workers
A single-vCPU worker has 900m to allocate after the kubelet
reservations, and the tenant addons' CPU requests use all of it. When
cert-manager's startupapicheck hook pod is the last one created it
stays Pending, the install times out, and the reinstall after
remediation runs into the same full node. Whether it is last depends
on install ordering, so the suite passed or failed on the race.

Two workers match the kubernetes suites' pool and leave room for the
hook pod.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-06 20:32:44 +03:00
Aleksei Sviridkin 9620be0103 feat(postgres): add per-user createRole and bypassRls attributes (#3505)
## What this PR does

- Adds `users.<name>.createRole` to the postgres chart, which sets the
`CREATEROLE` role attribute on a managed login user.
- Adds `users.<name>.bypassRls`, which sets `BYPASSRLS` on the same
user. The two attributes are needed together to provision a privileged
service role, see below.
- Sets `createrole_self_grant` as a role-level default alongside
`createRole`, without which the attribute is granted but not usable.
- Refuses `createRole` on PostgreSQL 15 and older, where it is an
escalation path rather than a bounded privilege.
- Makes the init job record the roles it manages in a superuser-only
marker instead of trusting role comments and refuses to adopt a role it
did not create once `createRole` has been used, see below.
- Adds `tests/user_createrole_test.yaml`: defaults, both explicit values
for each attribute, composition with `replication` and with each other,
the `createrole_self_grant` set and reset paths, and the version guard
on v15 and v13. Adds `tests/managed_roles_test.yaml` for the marker and
the adoption checks.

Today a tenant that needs to create its own roles has no way to do it
through the chart, because role attributes are not inherited through
role membership: granting `CREATEROLE` to the generated `<db>_admin`
group role has no effect on the users that are members of it, and
`<db>_admin` is `NOINHERIT` anyway. The attribute has to be set on the
login user itself, which only the init script can do. The remaining
option is to hand the tenant the `postgres` superuser, which also gives
it `COPY ... FROM PROGRAM`, arbitrary file access and command execution
inside the instance pod. These flags are the narrow version of that: the
tenant manages its own roles, and the superuser stays internal.

Both attributes are reconciled on every init-script run, like
`replication`, so setting one back to `false` emits the `NO...` form and
revokes it.

## Why `bypassRls` belongs in the same PR

`CREATEROLE` on its own does not let a tenant reproduce a role layout
that uses row-level security, which is the common case this feature
exists for. PostgreSQL only lets a role create other roles carrying
`BYPASSRLS` if it holds the attribute itself, so `CREATE ROLE
service_role ... BYPASSRLS` fails with `Only roles with the BYPASSRLS
attribute may create roles with the BYPASSRLS attribute`. Exposing
`createRole` without `bypassRls` therefore ships a feature that stops
one statement short of the goal.

`bypassRls` is not gated on a PostgreSQL version, because it carries no
escalation of its own: `BYPASSRLS` waives row-level security inside the
tenant's own databases, and the database owner already reads every row
in them. It grants no reach outside the tenant's data.

## Why `createrole_self_grant` is set with the attribute

This is the non-obvious half. PostgreSQL 16 grants a `CREATEROLE` holder
only `ADMIN OPTION` on the roles it creates, governed by
`createrole_self_grant`, whose default is empty. `ADMIN OPTION` allows
granting the role onward but not `SET ROLE` to it, and creating an
object owned by another role requires `SET`. The attribute therefore
looks functional -- `CREATE ROLE` succeeds -- while the very next
statement a tenant writes, `CREATE SCHEMA ... AUTHORIZATION <new role>`,
fails with `must be able to SET ROLE`. Every tenant granted `createRole`
would hit this and have to discover the parameter, so the chart sets it
as a role-level default and resets it when the attribute is withdrawn.

## Why the init job stops trusting role comments and role names

The init job runs as the superuser, and it trusted two things a
CREATEROLE holder controls for the roles it creates. It picked the users
and roles to drop by their comment, and on PostgreSQL 16 a CREATEROLE
holder may `COMMENT ON ROLE` every role it created, so it could hand the
chart for deletion a role the chart never created. It also adopted any
role that already existed under a declared name, so a role created under
a name an administrator declares later stayed under its creator's
control, for example `debezium` before `users.debezium.replication:
true`, or `newdb_admin` before `databases.newdb`.

Neither case can be caught from the catalog. Creating a role through an
intermediate role and then dropping that role leaves the squatted role
with no row in `pg_auth_members` at all, while the objects, role
settings and default privileges planted in it stay. On 17.11 such a
`debezium` had zero memberships, and after the chart adopted it
`migrator` could still run code as `debezium` with `REPLICATION` through
a function it had planted. The chart therefore keeps its own record:

- Chart-managed roles are members of `cozystack:managed`, a NOLOGIN role
that only the superuser can grant. On 16+ the membership is `WITH
INHERIT FALSE, SET FALSE`. The name is outside the username and database
charsets. The delete passes act only on these members. On its first run
the chart marks every role it had commented before, when no role could
yet hold CREATEROLE through it.
- Once any user has held `createRole`, recorded as
`cozystack.createrole_granted` on the `postgres` database (the same
mechanism as `cozystack.adopted_restore`), the chart refuses to adopt an
existing role it did not mark, both for users and for
`<db>_admin`/`<db>_readonly`. The init job fails with the role's name,
and a platform administrator must drop the role. Before that point
adoption works as before, so a fresh install that declares `users.app`
still adopts the role CNPG bootstraps.

## The PostgreSQL 16 floor on `createRole`

The floor is a security boundary, not a cosmetic check. Before
PostgreSQL 16, `CREATEROLE` allowed granting membership in any
non-superuser role, so a user holding it could grant itself
`pg_execute_server_program` and reach command execution on the instance,
which is exactly what not exposing the superuser is meant to prevent. I
verified both sides on a real cluster: on 15.15 `GRANT
pg_execute_server_program TO <createrole_user>` succeeds and `COPY ...
FROM PROGRAM 'id'` then returns `uid=26(postgres)`; on 16.11 and 18.1
the same grant is refused because it now requires ADMIN option on the
role. PostgreSQL 16 restricted `CREATEROLE` to roles the grantee
administers, which is what makes the attribute safe to expose. The guard
uses the same fail-fast `SECURITY:` idiom as the blocked
`postgresql.parameters` list in `templates/db.yaml`.

Note that a grant obtained on PostgreSQL 15 survives a major upgrade:
after moving an instance to 16.11 the user kept its membership in
`pg_execute_server_program` even though new attempts were refused.
Operators who exposed `CREATEROLE` out of band on an older major should
audit and revoke memberships in `pg_execute_server_program`,
`pg_read_server_files` and `pg_write_server_files` before relying on the
floor.

## Verification

The marker, adoption and `createRole` paths were rendered and run
against live PostgreSQL 13, 15, 16 and 17 containers. An upgrade from
the previous chart marks the roles it already managed and removes a
dropped user. What happens to a dropped role's objects comes from #4574
and is covered by its tests. A fresh install adopts the CNPG `app` role
in the same run that grants `createRole`. A forged comment is ignored,
and squatted `debezium` and `newdb_admin` roles stop the init job with
nothing granted to them. The orphaned-role pass still drops the roles of
a removed database, and every re-run is clean. Reverting each part of
the fix in turn fails `tests/managed_roles_test.yaml`.

Beyond the chart unit tests, I replayed a real migration checklist
against a live PostgreSQL 17.7 instance, connected as the managed login
user and never as the superuser: creation of four roles with their
attributes including `CREATEROLE` and `BYPASSRLS`, `CREATE SCHEMA ...
AUTHORIZATION` landing on the requested owner, trusted extensions into a
named schema, per-role and per-database `search_path`, per-role
`statement_timeout` and `idle_in_transaction_session_timeout`, RLS
including `FORCE ROW LEVEL SECURITY` verified end to end (a plain role
sees zero rows, the `BYPASSRLS` role sees them all), and `SECURITY
DEFINER` functions. All of it succeeds without the superuser. The one
thing that remains out of reach is `CREATE EVENT TRIGGER`, which
PostgreSQL restricts to superusers and no chart flag can change.

Other attributes and the predefined roles that a managed-service admin
role usually wants (`pg_monitor`, `pg_read_all_settings`, `pg_maintain`)
can follow the same per-user pattern in a later PR, minus the three that
grant OS access (`pg_execute_server_program`, `pg_read_server_files`,
`pg_write_server_files`).

### Screenshots

No UI change.

### Downstream repositories

`packages/apps/postgres/values.schema.json` gains two fields, which per
the trigger map reaches `cozystack/terraform-provider-cozystack`: its
postgres resource carries a hand-maintained schema plus an
expand/flatten pair for `users`, with no codegen from
`values.schema.json`, so `createRole` and `bypassRls` will be invisible
to Terraform users until they are added there by hand. I searched that
repository and found no open PR or issue covering it (the only open PR
is a dependabot Go module bump). I have not filed the follow-up, so I am
leaving the box unticked rather than claiming work that does not exist.
Happy to open the issue or the provider PR if a maintainer confirms that
is the preferred route.

I walked the rest of the map against the diff and found nothing else: no
package is added, renamed or removed, no version enum changes, no
default changes, `release.prefix` and the output Secret and Service
names are untouched, and no CRD outside the regenerated
`packages/system/postgres-rd/cozyrds/postgres.yaml` is affected. The
`README.md` parameter table is regenerated here and postgres is already
in the website generator's app list, so the website path stays
automated.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: not filed yet, see the note above
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(postgres): add `users.<name>.createRole` and `users.<name>.bypassRls`, which grant the CREATEROLE and BYPASSRLS attributes to a managed user so a tenant can create and manage its own roles, including roles that bypass row-level security, without the postgres superuser. `createRole` also sets `createrole_self_grant` on the user so that roles it creates can own objects. `createRole` requires PostgreSQL 16 or newer: on older majors CREATEROLE can be used to grant oneself pg_execute_server_program and reach command execution on the instance, so the chart refuses to render the attribute there. The init job now records the roles it manages as members of a superuser-only `cozystack:managed` role instead of trusting role comments. Once `createRole` has been used on an instance, it refuses to adopt an existing role it did not create.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* PostgreSQL user configuration now supports `createRole` and
`bypassRls` privileges.
* These privileges can be combined with replication access. Enabling
`createRole` also configures the role’s self-grant behavior.
* `createRole` requires PostgreSQL 16 or newer; configuring it on older
versions is rejected.

* **Documentation**
* Updated configuration schemas, examples, and guides to describe the
new options and PostgreSQL version requirements.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 20:03:34 +03:00
Aleksei Sviridkin f5325425d6 fix(vm-instance): supply cloud-init network-data to netplan guests (#4306)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A guest whose interfaces are rendered by netplan ignores two things the
platform already puts on the wire for it, and this adds an opt-in
`netplanDhcp` value that supplies the cloud-init network-data making it
ask for both.

`virt-launcher` starts a DHCP server on **every** bridge-bound
interface, secondary ones included — `newDHCPConfigurator` in
`pkg/network/setup/launcher/configurator.go` gates only on `iface.Bridge
!= nil`, and `BridgeConfigGenerator.Generate` serves the address the CNI
put on the pod-side interface along with that link's MTU. The chart
renders `bridge: {}` for every network, so a VM joined to a VPC subnet
is offered its reserved address and its MTU. cloud-init's fallback
network configuration covers a single interface, so on Debian and Ubuntu
nothing ever sends a DHCPDISCOVER on the second NIC and the interface
stays inert, while AlmaLinux (NetworkManager auto-profile) and Windows
come up. That asymmetry is the tell: with no DHCP server at all, none of
the three would work.

The same DHCP server advertises the launcher pod's search domains as
option 119, and `UseDomains=` defaults to `no` in systemd-networkd, so a
netplan-managed link keeps none of them. A cluster domain under `.local`
then reaches the mDNS exclusion in resolved and every lookup answers
`REFUSED`, which is #2085. `dhcp4-overrides: use-domains: true` is
enough on its own: the domains on the wire already carry whatever
cluster domain the cluster actually uses, so nothing needs to be
hardcoded here and #2372 does not collide with this.

### This is a regression, not a missing feature

`packages/apps/virtual-machine` carried exactly this, added in c63fb203b
(*[apps] virtual-machine: add vpc support*, 2025-10-21) as
`templates/secret-network-config.yaml` plus a `networkDataSecretRef`,
gated on `.Values.systemDisk.image`. 08a5f9890 (*feat(platform): remove
virtual-machine application*, 2026-02-12) deleted the application and
the template with it, and `vm-instance` never had one — `git log --all
-S networkDataSecretRef -- packages/apps/vm-instance/` is empty. The
gate could not be ported verbatim because `systemDisk.image` went away
in the same split, which is the likeliest reason nobody did. #2166 is
the other regression from that migration.

`docs/vm-external-vlan.md` has since documented the absence as a
property of the chart and prescribed a `write_files` + `netplan apply`
workaround, so this PR updates that page too.

### Why it is opt-in

The network-data renders only when `netplanDhcp` is `true`, and the
value defaults to `false`. Keying it on `instanceProfile` instead cannot
work: the profile defaults to `ubuntu` whatever image the disk holds,
and it is a KubeVirt preference rather than a declaration of the guest's
network stack. cloud-init passes a v2 configuration through verbatim
only for the netplan renderer; sysconfig, NetworkManager and eni turn
each `match` stanza into an interface literally named `cozy-en` or
`cozy-eth`, so a non-netplan guest handed this configuration loses DHCP
on its real NIC.

Two cases where the value must stay off, both stated in its description
and in `docs/vm-external-vlan.md`:

- A guest not rendered by netplan (AlmaLinux, Rocky, Alpine), for the
reason above.
- A guest that configures a NIC statically through netplan, as the
external-VLAN recipe does. Netplan writes one networkd file per
definition ID and systemd-networkd applies only the first `.network`
file in lexical order that matches a link, so
`10-netplan-cozy-en.network` shadows `10-netplan-enp2s0.network` and the
static address is never applied, whatever number the user's netplan file
carries. Reproduced with `netplan generate` on Ubuntu 24.04.

Matching by MAC with a v1 configuration would be renderer-independent
and would not shadow a named interface, but it needs the chart to know
the MACs at render time; `lookup` cannot supply them on a first install,
so the chart would have to pin them, which is a larger change that
overlaps #4038.

### What this does not do

A VM that leaves `netplanDhcp` unset renders exactly as before: no
network-data Secret, and no cloud-init disk where there was none.
Turning the value on for a VM with neither `cloudInit` nor `sshKeys`
adds a cloud-init disk, and because KubeVirt uses the pinned firmware
UUID as the NoCloud instance-id, a guest that has never seen that id
treats its next boot as a first boot (new SSH host keys); the value's
description says so. On a VM that already has a cloud-init disk the
instance-id does not change, and it is derived from the namespace, the
name and `cloudInitSeed`, so recreating the VM keeps it too; cloud-init
applies network configuration per instance by default, so such a guest
picks the network-data up once `cloudInitSeed` is changed. Static
addressing is still `userData`.

### Tests

`tests/network_data_test.yaml`, 6 cases: the rendered network-data
asserted in full (both properties, not just the presence of the key),
the VM carrying network-data with no user-data requested, user-data and
network-data side by side, a VM on the default profile rendering neither
the Secret nor a cloud-init disk, user-data alone when the value is off,
and the value rather than the profile deciding. With the templates gated
on the `ubuntu` profile again, the default-profile case and the
value-off case go red.

`helm unittest` green on all four suites of the package (32 tests).
`make generate` output is committed (`values.schema.json`, `README.md`,
`api/apps/v1alpha1/vminstance/types.go`,
`packages/system/vm-instance-rd/`). `helm lint` fails on this package
identically on an untouched tree (`_cluster.scheduling` nil pointer,
`cozy-lib` dependency metadata), so it is not a signal here.

### Screenshots

Not a UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

<!-- Trigger map walked file by file against the diff.
`docs/vm-external-vlan.md` documents the absence this PR removes, which
is the "make a documented workaround obsolete" trigger, so the website
copies of that page need the same edit and issue 702 carries it.
`netplanDhcp` is a new field in `values.schema.json`, which the provider
models by hand, and issue 62 there carries it. No package added, renamed
or removed; no hack/ or ApplicationDefinition change; the cozy-proxy
annotations in templates/service.yaml are untouched. -->

- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/702
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/62
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(vm-instance): a new `netplanDhcp` value, off by default, makes the chart supply cloud-init network-data that requests DHCP on every ethernet interface and accepts the DNS search domains offered over DHCP. A netplan guest (Ubuntu, Debian) attached to a VPC subnet then configures that interface on its own instead of leaving it unaddressed, and cluster service names under the cluster domain resolve inside the guest. Leave it off for guests not rendered by netplan and for guests that configure a NIC statically through netplan. VMs that do not set it are unchanged.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an opt-in setting to request DHCP on Ethernet interfaces in
supported virtual machines while retaining DHCP-provided DNS search
domains.
* The setting is off by default. Enabling it can add a cloud-init disk
and may interfere with static network settings or guests that do not use
netplan.

* **Documentation**
* Updated setup guidance with considerations for enabling DHCP and
configuring static networking on VLANs without a DHCP server.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 19:48:34 +03:00
Aleksei Sviridkin bcda5c893a feat(kubevirt)!: drop HotplugVolumes for declarative volume hotplug (#4453)
## What this PR does

This turns off the deprecated KubeVirt `HotplugVolumes` gate and moves
the platform to declarative volume hotplug. It changes how every volume
hotplug on the platform works, so the title carries `!`. Closes #4032.

This PR is stacked on #4451 and has to merge after it. #4451 adds the
kubevirt-csi flag this PR flips.

The replacement gate is already on in KubeVirt 1.9, and the old gate is
the only thing keeping it off. `DeclarativeHotplugVolumes` is Beta at
v1.9.0, and Beta gates are on by default. virt-controller skips
`handleDeclarativeVolumeHotplug` while `HotplugVolumes` is on, so
dropping the old gate is what switches the new one on.

The kubevirt-csi fallback goes off in the same change. With the old gate
off, virt-api rejects the VMI-level `removevolume` on a VM-owned VMI, so
the chart now passes `--enable-vmi-hotplug-fallback=false`. The comments
next to the gate and the flag point at each other.

This needs the rebuilt kubevirt-csi image. A driver built before #4451
does not know the flag and exits on it at startup. The image is stamped
at release preparation and ships in the same release as this change, so
only a `make apply` from main before then would crash-loop the CSI
controller.

What changes at runtime on an upgrade:

- The VM-level `addvolume` and `removevolume` calls now edit `VM.spec`
directly. virt-controller then makes the VMI hotplug volumes match it,
instead of going through `VM.status.volumeRequests`.
- Volumes hot-plugged through the VM subresource are in `VM.spec`, so
they stay attached.
- During a tenant teardown, when the VM goes before its VMI, the driver
no longer calls the VMI subresource. It waits for the VMI status to drop
the volume, and the garbage collector deletes the VMI with its VM.
Upstream waits up to two minutes. If the VMI is still there after that,
the unpublish fails and is retried.

A new platform migration keeps old VMI-only volumes attached. On its
first pass virt-controller removes every hotpluggable VMI volume that is
missing from `VM.spec`. Until July 2025 kubevirt-csi attached volumes at
the VMI level (the Role then granted
`virtualmachineinstances/addvolume`), and live migration carries such a
VMI across KubeVirt updates. Without help, a tenant node not recreated
since then would lose a mounted disk on upgrade.

Migration 59 copies each such volume and its disk from the VMI into the
VM template before anything else changes. Adding a hotplug volume to the
VM spec is a valid live update in KubeVirt, so no VM needs a restart. It
runs as the platform chart's pre-upgrade hook, before the upgrade
re-points `cozystack-packages`. The KubeVirt CR and the tenant
kubernetes charts both render from that source, so the gate and the flag
cannot change before the migration has finished. It only touches VMs
that need it, logs each one, and a second run changes nothing.

The migration also makes ephemeral hotplugs permanent. A volume a user
added at the VMI level, for example with `virtctl addvolume` without
`--persist` from an older virtctl, has the same shape as kubevirt-csi's
old attachments, because virt-api builds both the same way. There is
nothing reliable to tell them apart by, so both are copied. Leaving the
user's volume out would not keep it ephemeral: the first reconcile would
unplug it from the running VM. After the upgrade it is part of `VM.spec`
and attaches again on a restart, and `virtctl removevolume` drops it.

Ephemeral hotplug stops working on VMs. With the old gate off, virt-api
refuses VMI-level `addvolume` and `removevolume` on a VMI that a VM owns
(`vmiVolumePatch` in v1.9.0). A volume on such a VMI is added and
removed through the VM from then on. A VMI without a VM is not affected.

Only running VMIs are copied. virt-controller reconciles no other VMI,
and a stopped VM builds its next VMI from `VM.spec`, so copying from a
stopped one would bring back volumes the restart was going to drop.

Migrations run only on the isp-full, isp-full-generic and isp-hosted
variants. KubeVirt comes from the platform only on isp-full and
isp-full-generic, so those are covered. On the default variant, where
packages are installed by hand, nothing runs the migration.

This optional check lists the VMIs and volumes the migration would copy.
It uses the same rule. On the default variant, run it before upgrading
and recreate each tenant node it lists:

```bash
kubectl get vm,vmi --all-namespaces --output json | jq --raw-output '
  ([.items[] | select(.kind == "VirtualMachine")
    | {key: "\(.metadata.namespace)/\(.metadata.name)", value: [.spec.template.spec.volumes[]?.name]}]
    | from_entries) as $vm
  | .items[] | select(.kind == "VirtualMachineInstance")
  | ([.metadata.ownerReferences[]? | select(.controller == true and .kind == "VirtualMachine" and (.apiVersion | startswith("kubevirt.io/"))) | .name][0]) as $owner
  | "\(.metadata.namespace)/\($owner)" as $key
  | select(.status.phase == "Running" and $owner != null and $vm[$key] != null)
  | .spec.volumes[]? | select(.dataVolume.hotpluggable == true or .persistentVolumeClaim.hotpluggable == true)
  | .name as $n | select($vm[$key] | index($n) | not)
  | "\($key) \($n)"'
```

Empty output means nothing is affected. I checked the query against the
migration's test fixtures only. Neither the query nor the migration has
run on a live cluster.

#4287 also adds a migration 59. If it merges first, this one moves to
60: the file, the version it stamps, `migrations.targetVersion`, the
bats file (its `run_migration` argument and expected `STAMP`) and
`hack/testdata/migration-59-vmi-hotplug`.

A VM with a pending hotplug request at upgrade time may also get stuck.
virt-controller still processes `VM.status.volumeRequests` before the
declarative handler, and it applies them through the VMI subresource,
which virt-api rejects once the old gate is off. The sync then fails
before the VM spec is updated, so nothing clears the request, and every
later sync of that VM fails the same way. A retry through the VM
subresource edits the spec and clears it, which covers kubevirt-csi's
own requests. A one-shot request stays stuck until the VM restarts. I
read this from the v1.9.0 code and haven't reproduced it. Waiting until
this lists nothing before upgrading narrows the window, but a request
made during the rollout can still hit it:

```bash
kubectl get vm --all-namespaces --output json | jq --raw-output '.items[] | select((.status.volumeRequests // []) | length > 0) | .metadata.namespace + "/" + .metadata.name'
```

The two halves live in different HelmReleases, so there is a short
window during the rollout where the driver and the gate disagree. In
that window an unpublish fails and is retried until both are updated.

A tenant cluster whose `kubernetes` HelmRelease is suspended or failing
keeps its old driver for longer. Drivers from before July 2025 attach
through the VMI subresource, so on such a tenant every attach is refused
until its release reconciles. Such a driver can also attach a volume
after the migration has run and before the gate flips. The migration
doesn't see that volume, and the first declarative reconcile unplugs it
from the running VM.

The migration lists the VMs once and patches each one with a
`resourceVersion` test, so a VM that changes in between fails the hook
Job. The Job has `backoffLimit: 3`, and each retry lists again and skips
the VMs already patched. Re-reading only the conflicting VM would avoid
the retry; I left that out.

The new gate is listed in the CR even though it is on by default. With
the old gate gone it is the only hotplug path, and the chart refuses a
render that disables it. I found no other user of `HotplugVolumes` in
the platform or in the downstream repositories.

The VMI-level fallback may have been failing already. The kcsi Role has
no rule for the VMI-level `removevolume` (#4452), so that call likely
got 403 with or without the gate. I haven't checked that on a cluster.

`hack/migration-59-vmi-hotplug-volumes.bats` runs the migration against
a fake kubectl: a VM that needs the copy, VMs and VMIs that must be left
alone (a VMI that is not Running among them), a second run, a cluster
without KubeVirt and the fail-closed paths. The helm-unittests for the
KubeVirt CR pin the whole gate list, accept `HotplugVolumes` in
`disabledFeatureGates` and refuse `DeclarativeHotplugVolumes` there. The
CSI test pins the flag at false, and the `make update` idempotency test
still passes. None of that exercises hotplug itself. The real test is
e2e with volume attach and detach on a tenant cluster, including a
teardown. No CI lane upgrades from the previous release yet, so an
upgrade with a live VMI-only volume is untested; #4463 tracks that test.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kubevirt)!: KubeVirt now runs with DeclarativeHotplugVolumes instead of the deprecated HotplugVolumes feature gate, and kubevirt-csi no longer uses the VMI-level hot-unplug. Volume hotplug requests are applied through VM.spec. The upgrade copies volumes that older kubevirt-csi versions attached only to the VMI into VM.spec, so they stay attached; the check in the PR description is optional. This also applies to ephemeral volumes added to running user VMs at the VMI level, such as with virtctl addvolume without --persist: they become part of VM.spec and attach again after a VM restart. virt-api now refuses VMI-level addvolume and removevolume on a VMI owned by a VM, so such volumes are added and removed through the VM. A tenant cluster whose kubernetes release is suspended or failing keeps its old kubevirt-csi, and a driver from before July 2025 cannot attach volumes there until that release reconciles, and a volume it attaches during the upgrade itself can be unplugged again. A values override that puts DeclarativeHotplugVolumes into disabledFeatureGates now fails the kubevirt render. Clusters on the default variant do not run platform migrations and should run that check before upgrading.
```





<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
  * Declarative hotplug volumes are enabled by default for KubeVirt.
* Running virtual machines keep eligible hotplugged volumes and disks
when their configuration is migrated.
* **Bug Fixes**
* The migration avoids adding duplicate volumes or disks and does not
mark the migration complete if required operations fail.
  * The Kubernetes CSI driver no longer uses the VMI hotplug fallback.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 17:06:03 +03:00
Aleksei Sviridkin ff3c409881 fix(backups): refuse a foreign -orig PVC and record the kept disk's reclaim policy (#4671)
## What this PR does

Two defects in `renamePVC`
(`internal/backupcontroller/velerostrategy_controller.go`), the step of
an in-place VMInstance restore that keeps each disk as
`<dv>-orig-<shortHash(RestoreJob name)>` when `keepOriginalPVC` is true,
the default.

### A taken `-orig` name skipped the rename, and the restore went on

`renamePVC` returned `nil` as soon as a PVC with the `-orig` name
existed (`main:1263-1266`), whatever that PVC was. `prepareForRestore`
then reached Step 4 (`main:1172-1183`), which deletes the DataVolume of
every disk regardless of what Step 3 did, because a Step 3 error was
only ever turned into a Warning event (`main:1165-1168`). The restore
therefore ran exactly as `keepOriginalPVC: false` does and reported
`Succeeded`: the live disk was not kept, and the `-orig` name held a
different, older disk that the user takes for the pre-restore copy.

When that name can already exist:

- A RestoreJob created under the name of a deleted one. The hash covers
only the RestoreJob name, and nothing in the controller ever deletes an
`-orig` PVC, so the predecessor's PVC is still there. This is the
realistic case, and it is what a scripted restore with a fixed
RestoreJob name runs into on its second run unless the `-orig` PVC was
removed in between.
- Two RestoreJobs with different names whose 4-hex-digit hashes collide
(16 bits) while restoring the same disk in the same namespace. Rare, but
nothing guards it.
- A reconcile of the same RestoreJob after a controller restart in the
middle of a rename. The name is ours, but the early return treated a
half-done rename as complete: the original PVC stayed in place and the
PV stayed bound to it.
- Truncation does not apply: the name is not truncated, and one longer
than 253 characters makes the `Create` fail, an error that was swallowed
in the same way.

Each `-orig` PVC now carries the UID of the RestoreJob that created it,
in `backups.cozystack.io/restore-job-uid`. `prepareForRestore` checks
every `-orig` name before it suspends or halts anything, and fails the
RestoreJob, naming the PVC, when any of them is held by a PVC another
RestoreJob created. When the PVC is ours, `renamePVC` resumes: it
deletes the original PVC only while it still claims the kept PV, and
repoints the PV unless it already points at the `-orig` PVC, so a rerun
is a no-op. Any other rename error now holds the restore back before
Step 4 deletes a DataVolume, and the restore is retried, instead of the
error becoming an event (see the next section).

An `-orig` PVC created by an earlier controller has no UID annotation
and is refused like any foreign PVC. The only RestoreJob that loses by
this is one whose controller was upgraded after its rename and before
its Velero Restore was created; it fails with the message above rather
than going on.

### The rename read its own write back through the cache

`renamePVC` created the `-orig` PVC, then read it back through the
cached client to learn its UID for the PV's claimRef (`main:1314-1316`).
The informer often has not seen a PVC created a moment earlier. On our
lab, running Cozystack v1.6.3, an in-place restore of a two-disk VM with
`keepOriginalPVC: true` renamed and restored the system disk, while the
data disk lost the race, with this event on the RestoreJob: `Failed to
keep old PVC vm-disk-e2e-dc-fix-data: failed to get -orig PVC
vm-disk-e2e-dc-fix-data-orig-a574: PersistentVolumeClaim
"vm-disk-e2e-dc-fix-data-orig-a574" not found`. As the error was only an
event, the data disk's DataVolume was deleted and its original PVC
stayed in place. Velero skipped that PVC and created no DataDownload for
it, so its content was not restored, yet the restore finished
`Succeeded`. The `-orig` PVC was left `Pending` on the same PV, which
was by then `Retain`.

`Create` already fills in the UID, so the read-back is gone. The other
reads the rename decides on (the `-orig` PVC, the original PVC and the
PV, before and after its own writes) go through the manager's uncached
reader, wired as `APIReader`. The PV rebind retries on conflict, since
the PV controller also writes the PV once the original PVC is deleted. A
rename that fails for any reason other than a foreign `-orig` name
requeues the restore rather than failing it. The DataVolume is still
there, and the retry finds its own `-orig` PVC by the RestoreJob UID and
resumes where it stopped. A cause that never clears keeps the RestoreJob
`Running`, with a Warning event on each attempt.

#4673 adds the same `APIReader` field and `apiReader()` helper to
`RestoreJobReconciler`, for its finished-job guard. Whichever of the two
lands second drops its own copy when it rebases.

### The PV's reclaim policy was overwritten without being recorded

`renamePVC` switches the PV to `Retain` so it survives the deletion of
the original PVC (`main:1289-1293`), and the policy it had was lost. I
looked for a path in the controller that releases or cleans up `-orig`
PVCs, and there is none: `cleanupOnDelete` on a Velero RestoreJob
removes only the Velero `Restore` and the resource-modifier ConfigMaps,
and no code deletes an `-orig` PVC or touches its PV again. A PV that
was `Delete` therefore stays `Retain` for good, and its volume leaks
when an operator deletes the `-orig` PVC.

The original policy is now recorded in
`backups.cozystack.io/original-reclaim-policy`, on the PV and on the
`-orig` PVC, following the `backups.cozystack.io/` prefix the controller
already uses for `underlying-resources`. A value already present on the
PV is kept, as it predates any rename. Since there is no release path to
restore it from, `examples/backups/vmi/scenario-user-restore.md` gains a
section for the operator who deletes an `-orig` PVC: set the recorded
policy back on the PV first, with a snippet that leaves a PV without the
annotation on `Retain`.

Restoring the policy from the controller itself, for instance once the
Velero Restore has completed and the PV is `Bound` to the `-orig` PVC,
would close the leak without the operator. I did not do it, because it
changes what deleting an `-orig` PVC destroys, from nothing to the
volume; that is a maintainer call, and the annotation is what it would
read.

### How this fits with #3427

#3427 is complementary, and the two do not touch the same code: it
changes `collectUnderlyingResources`, this PR changes
`prepareForRestore` and `renamePVC`. Both append tests to
`velerostrategy_controller_test.go`, so whichever lands second may need
a trivial rebase.

- #3427 makes a standalone VMDisk report its own disk, which sends
VMDisk in-place restores through this same `renamePVC`. Once both land,
the VMDisk path gets the same guard and the same annotation.
- Its `examples/backups/{vmi,vmdisk}/cleanup.sh` and the Chainsaw
`finally` reclaim each `-orig` PVC by patching its PV to `Delete` before
deleting the PVC. That works around, in the e2e teardown, the leak
described above. It is correct on the LINSTOR classes, which are
`Delete`, but it would destroy a volume whose class was `Retain`. With
this PR the scripts can read
`backups.cozystack.io/original-reclaim-policy` instead of hard-coding
`Delete`. I have not changed them here, since they only exist on that
branch.
- Its round-trips use fixed RestoreJob names and assert that a
`<pvc>-orig-*` PVC exists after the restore. If a failed teardown leaves
one behind, today the next run skips the rename and the assertion still
passes on the stale PVC; with this PR the RestoreJob fails and names it.
- Its cleanup comments describe the `-orig` PVC as a bare ObjectMeta
with no labels or owner. That stays true: this PR adds annotations only,
so its name-based lookup is unaffected.

### Tests

New tests in
`internal/backupcontroller/velerostrategy_controller_test.go`, run
through `prepareForRestore` with the DataVolume in the dynamic fake
client:

- `TestPrepareForRestore_KeepOriginalPVC_RefusesForeignOrigPVC`: an
`-orig` PVC without the annotation, and one carrying another
RestoreJob's UID, both fail the restore with the PVC named, and leave
the DataVolume, the live PVC, the PV and the foreign PVC untouched.
- `TestPrepareForRestore_KeepOriginalPVC_ResumesOwnRename`: a rename
interrupted after the `-orig` PVC was created completes on the next
pass, and a second pass changes nothing.
- `TestPrepareForRestore_KeepOriginalPVC_FailedRenameKeepsDataVolume`: a
rename that fails for another reason (missing PV) requeues the restore
without failing it and keeps the DataVolume, and once the PV exists the
retry completes the rename.
- `TestPrepareForRestore_KeepOriginalPVC_CacheLagsBehindWrites`: the
reconciler's cached client serves every read from the objects as they
were seeded, through an `interceptor` over the fake client, as an
informer that has not caught up with this reconcile's writes would,
while `APIReader` sees them. The rename must complete.
- `TestRenamePVC_RecordsOriginalReclaimPolicy`: a `Delete` PV and a
`Retain` PV both end up `Retain`, with their original policy recorded on
the PV and on the `-orig` PVC.

With the controller reverted to `main` (only the two annotation
constants kept so the tests compile), all four fail:
`prepareForRestore() = ready true, nil error` with `DataVolume was
deleted although the disk could not be kept` for both foreign cases and
for the failed rename, `original PVC still exists` and the PV still
bound to it for the interrupted rename, and an empty
`original-reclaim-policy` on the PV and on the PVC. For the cache race,
the first commit alone fails `CacheLagsBehindWrites` with the lab's
error, `failed to keep PVC vm-disk-ubuntu-source: failed to get -orig
PVC vm-disk-ubuntu-source-orig-de3e: persistentvolumeclaims
"vm-disk-ubuntu-source-orig-de3e" not found`, and
`FailedRenameKeepsDataVolume` with a terminal error instead of a
requeue. On `main` the race test reproduces the lab outcome: `original
PVC still exists`, and the PV still claimed by it.

With both commits, `go test -count=1 ./internal/backupcontroller/...`
passes.

No API type, CRD, chart or values file changes, so there is nothing to
regenerate.

### Screenshots

N/A, no UI change.

### Downstream repositories

Walked the trigger map against the diff: the restore preparation in
`internal/backupcontroller/velerostrategy_controller.go`, its tests, and
`examples/backups/vmi/scenario-user-restore.md`. No CRD,
ApplicationDefinition, values, schema, enum, namespace, label, metric or
tooling change, so terraform-provider-cozystack, ansible-cozystack, ccp,
talm, cozyhr, cozy-proxy, cozystack-telemetry-server,
external-apps-example, examples and community are not reached.
`RestoreJob.spec.options` stays opaque to the provider.

cozystack/website is reached:
`content/en/docs/next/virtualization/backup-and-recovery.md` (and each
released copy) tells users to delete the `*-orig-*` PVCs "to free up
storage". A PV left on `Retain` frees nothing, and the page should
explain the recorded policy and the new failure for a reused RestoreJob
name. No follow-up exists there yet, so the box stays empty until one is
filed and linked.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): an in-place VM restore with keepOriginalPVC now fails, before the VM is halted, when the <disk>-orig-<hash> PVC it would keep the disk under already exists and was not created by this RestoreJob (for instance one left by a deleted RestoreJob of the same name), instead of skipping the rename and restoring without keeping the current disk. Any other failure to keep a disk, including a race in which the rename read its just-created -orig PVC back through the cache and failed, now holds the restore back and retries it before its DataVolume is deleted. Previously such a restore reported success without restoring the disk. The PV's reclaim policy before the switch to Retain is recorded in the backups.cozystack.io/original-reclaim-policy annotation on the PV and the -orig PVC; set it back on the PV before deleting the -orig PVC, or its volume is not reclaimed.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* In-place restores that preserve original PVCs can now resume safely
after interruptions, including when reads lag behind recent changes.
* Restores stop before suspending the VM if a conflicting `-orig` PVC
already exists. PVC rename failures can be retried, while conflicting
names remain fatal.
* Original PV reclaim policies are recorded and restored before
preserved PVCs are deleted.
* **Documentation**
* Updated the restore guide with disk-reclamation steps and guidance for
resolving leftover PVC name conflicts.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 16:51:49 +03:00
IvanHunters f05718b2de feat(redis)!: refuse external access without TLS at admission (#4639)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

An admission policy now refuses a Redis release that would expose a
plaintext endpoint outside the cluster. Redis never turns TLS on by
itself, so `external: true` without `tls.enabled: true` is exactly that
case. The error names the release and a remedy that works.

The rule has three parts, because TLS is fixed when a Redis instance is
created: the RedisFailover schema refuses a change, while the Redis API
accepts one and the HelmRelease then fails. A spec's `tls.enabled` is
therefore a request, not proof of what the instance runs.

- On create, `external: true` needs `tls.enabled: true`.
- On update, turning `external` on is refused whatever `tls.enabled`
says.
- While `external` stays on, `tls.enabled` cannot change. Before the
first reconcile nothing else stops TLS being dropped from a release
admitted as external with TLS. Withdrawing `external` in the same edit
lifts the freeze, and the callers trusted by
`cozystack-tenant-host-policy` are exempt, so an instance whose TLS was
flipped earlier can still be unblocked.

Edits to a release that was already external pass, so releases exposed
in plaintext before this policy keep working. The guarantee covers the
Redis app. A generic publisher such as `tcp-balancer` can still forward
to a plaintext Redis, by design.

This diverges from the Security section of the
`external-database-exposure` proposal, which refuses only an explicit
`tls.enabled: false` and covers every engine. That rule was written
before redis had TLS and matches only postgres. Postgres is left out
here on purpose: CloudNativePG keeps `ssl=on` and its default pg_hba
accepts plaintext from any client that asks, whatever the chart's
`tls.enabled` says, so an admission rule on that field closes nothing.
That gap is #4619. The proposal update and a decision record are
cozystack/community#91.

The policy follows `cozystack-tenant-host-policy`: a
ValidatingAdmissionPolicy in `cozystack-basics`, enforced inside
`cozystack-api` for the aggregated kind. Its decisions are tested in Go
by compiling the shipped template with the apiserver's own CEL compiler
and running Redis objects through it, including the bypass sequences
found in review. A structural chart test can't show what the policy
admits. The redis README, the `external` field description and the
threat model describe the new rule.

Related: #4622 (valkey `external` is always plaintext), #4624 (the redis
`tls.enabled` description implies a live switch).

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

I walked the trigger map against the diff and nothing in it fires, but I
left the boxes empty because cozystack/community is reached anyway: the
`external-database-exposure` proposal still states the old admission
rule. Its update and a decision record are cozystack/community#91.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(redis)!: a Redis release can be exposed outside the cluster only when it is created with `tls.enabled: true`. Creating one with `external: true` and no TLS, turning `external` on for an existing instance, or changing `tls.enabled` while `external` is on is refused at admission. Instances already external without TLS keep working and stay editable.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Redis admission checks now require TLS when external access is enabled
at instance creation. They block enabling external access on an existing
instance and changing TLS while external access remains enabled.
Disabling external access in the same update allows TLS changes.
* Existing instances that already had external access without TLS before
the policy was introduced remain editable.
* **Documentation**
* Updated Redis configuration and security guidance to explain these
restrictions and TLS migration behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 16:39:23 +03:00
Aleksei Sviridkin ae738e6047 test(backups): pass the flavor in the major-version test calls
Two tests that came with the PostgreSQL major-version check call
createCNPGBackupArtifact and marshalCNPGBackupSnapshot with the
argument lists from before the flavor parameter was added, so the
backupcontroller test package does not compile on main. Both calls
take the default flavor the other tests in the file pass.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-06 16:39:12 +03:00
IvanHunters d367846030 test(e2e): cover the computeplane module end to end (#4713)
## What this PR does

This adds an e2e suite for the ComputePlane tenant module, which had
only render tests so far. The suite lives in
`hack/e2e-chainsaw/computeplane/`. It turns the module on in its own
tenant, `tenant-cplane`, so the shared `tenant-test` is never toggled.
That tenant is a child of `tenant-root`, not of `tenant-test`, because a
tenant two levels down cannot reach the etcd it inherits from root
(#4726).

What it checks on a real cluster:

- the module release installs, and the cluster release it renders keeps
the pinned addons and the `internal.cozystack.io/computeplane` label;
- the pool release gets only the keys from the module's per-pool
allowlist (the patch sends a `nameOverride` through the ComputePlane API
and checks it is dropped);
- Kamaji writes `computeplane-cluster-admin-kubeconfig`, and the tenant
can't get at it: the Secret is not labelled `tenantresource=true`,
`tenantsecrets` does not list it to the tenant's super-admin group, and
`kubectl auth can-i get` answers `no` for that group (the read was
possible until #4722);
- cert-manager and ingress-nginx come up inside the cluster, and the
Proxied ingress Service gets a ready endpoint;
- the module and cluster releases have no remediation cycle in
`status.history`;
- on teardown, the in-cluster addon releases are gone before the
KamajiControlPlane starts deleting;
- turning the module off removes all of it: releases, in-cluster addons,
control plane, Secret, worker VMs and disks, and the LINSTOR pools go
back to their pre-suite level.

The issue names the Secret `computeplane-admin-kubeconfig`. The module
writes `computeplane-cluster-admin-kubeconfig`, so the suite checks that
name.

The issue also lists fluxcd as a pinned addon. That is out of date. The
module runs no in-cluster Flux on purpose
(`packages/extra/computeplane/templates/cluster.yaml:108-110`), so the
suite checks cert-manager and ingress-nginx only.

One part of the issue is not covered here. The `placement: ComputePlane`
routing in cozystack-api doesn't exist yet, so there is no catalog app
to run on the cluster. The issue ties that to "together with, or right
after" this test, so I left it for the routing work, tracked in #4715.

Things the review found outside this diff, each filed on its own: the
module release's 10m install timeout against a slow cluster bringup
(#4714), the module docs not matching the per-pool allowlist (#4716),
and the LINSTOR helpers in `_lib` running `kubectl exec` without a
timeout (#4111).

The suite brings its own pool: `md0`, one worker, same sizing as the
kubernetes suites. The default `md0` that the module renders for empty
`nodeGroups` asks for the `replicated` StorageClass. The container lane
doesn't have it, so on this lane the default pool stays covered by the
render tests only.

ComputePlane workers can't use the in-sandbox Talos image cache. The
pool allowlist drops `talos`, so the worker pulls its disk from the
public Image Factory (#4700). If the worker join or an addon release
times out, the catch block prints the worker image source and points at
that issue, so a red run caused by the factory is easy to tell apart.

The suite itself needs no entry in `hack/select-e2e.sh`, because
`computeplane-application` maps to it by name. That source depends on
`kubernetes-application` and re-sources the kubernetes charts. So PRs
that touch the kubernetes charts or ingress-nginx now select
`computeplane` too, which I estimate adds 12-15 minutes to a green run
(kubernetes-previous takes about 8 minutes with two workers, plus
teardown). A red run can take much longer, because the waits sit above
the release install timeouts. The suite header has the numbers. It is
the third suite that brings up a tenant cluster, and the 215m E2E job
cap is sized for the red path of the kubernetes pair only. If worker
join breaks everywhere, all three go red and can run past the cap, which
cancels the job and loses its log. I left that as an accepted risk.
`hack/select-e2e_test.bats` now expects `computeplane` in the
ingress-nginx selection.

The computeplane source also owns `apps/kubernetes-nodes`. Before this
PR a pool chart edit reached no suite and ran everything. With the new
suite alone it would have selected only `computeplane`, which never
creates `KubernetesNodes` natively. So `kubernetes-nodes-application`
now maps to `kubernetes-latest kubernetes-previous computeplane` in
`hack/select-e2e.sh`, and two new tests pin that for a pool chart edit
and a `kubernetes-nodes-rd` edit.

The suite is not in the leak-guard allowlist. Its teardown already fails
when a TenantControlPlane sticks on its finalizer (#4489), because the
admin kubeconfig Secret it owns stays behind, so an exemption would only
hide other leaks. The README suite list now names the suite.

JFYI, not a defect claim: the Proxied ingress Service from the
kubernetes chart has `internal.cozystack.io/tenantresource: "true"`. So
the tenant sees it, even though `computeplane-rd` has `services.include:
[]`. The module doesn't promise to hide it.

How I checked it. `chainsaw lint` passes at v0.2.15, the version the
sandbox pins. I also ran the suite against an envtest apiserver, with
fake controllers applying the real `helm template` output of the tenant
and computeplane charts. The intact run is green. Each of these
breakages turns it red at the step meant to catch it: label removed,
addon `exposeMethod` changed, a key outside the pool allowlist, the
allowlist `pick` replaced by the whole group, stale Ready from an older
generation, wrong Secret name, Secret labelled `tenantresource`, Secret
listed in `tenantsecrets`, a direct RBAC grant on the Secret,
ingress-nginx addon not Ready, no ready ingress endpoint, an
`uninstalled` history entry, the control plane deleted before the addon
releases, an addon release or worker VM left after teardown.

Closes #3369
Closes #4747

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
test(e2e): add an end-to-end suite for the ComputePlane tenant module
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
* Added end-to-end coverage for enabling and disabling the ComputePlane
tenant module, including generated releases, pool value filtering, admin
credential access, addon and cluster readiness, and resource cleanup.
* Kubernetes nodes changes now select the ComputePlane suite alongside
the latest and previous Kubernetes suites.

* **Documentation**
* Added the ComputePlane suite to the documented end-to-end test suite
list.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 16:37:26 +03:00
Aleksei Sviridkin 3fc45ed183 fix(backups, postgres): refuse a restore into another PostgreSQL major (#4669)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A CNPG restore is physical, so the recovered data files only open on the
PostgreSQL major version that wrote them. `reconcileCNPGRestore` purged
the target Cluster and its PVCs without comparing the two. Observed on
v1.6.3: a PostgreSQL 18 backup restored into a v16 application emptied
the target, the `full-recovery` pods then looped on `FATAL: database
files are incompatible with server`, and the RestoreJob failed at the
30-minute deadline with the target left empty. An in-place restore into
an application whose `spec.version` was raised since the backup ends the
same way.

The restore now compares the two majors before the purge, ahead of the
WAL-archive gate, and when they differ marks the RestoreJob `Failed`
with reason `MajorVersionMismatch` and a message naming both versions.
Nothing is suspended, patched or deleted on that path.

- Target major: the Postgres application's `spec.version` (`v13` to
`v18`). That is what the chart renders the recovered Cluster from once
the live one is gone, so the live Cluster's image would be the wrong
thing to compare.
- Backup major: the `cnpg.io/Backup`'s `status.majorVersion`, now copied
into the artifact's `driverMetadata` as `cnpg.io/major-version` at
backup time, because barman retention deletes the `cnpg.io/Backup` while
the artifact stays restorable. An artifact taken before this change
falls back to the `cnpg.io/Backup` while it still exists.
- When either side is unknown (no record and the `cnpg.io/Backup` gone,
a CNPG that predates `majorVersion`, an empty `spec.version`), the
restore goes ahead as before and the controller logs it, so existing
backups keep restoring.

`TestReconcileCNPGRestore_RefusesCrossMajorRestore` drives the real
reconcile against a fake client with a live Cluster and PVC on the
target and counts every `Delete`. With the check removed, the two
refusal cases (18 into a v16 copy, 16 in place into an app moved to v18)
and the fallback case fail: the Cluster and the PVC are deleted and the
app is patched for recovery. With the check in place they end `Failed`
with zero deletes, while the same-major and unknown-version cases still
reach the purge. `TestCreateCNPGBackupArtifact_RecordsMajorVersion`
covers the `driverMetadata` record.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Walked against the diff: it touches only `internal/backupcontroller`,
with no CRD, no app `values.yaml` or schema and no ApplicationDefinition
change. The new `cnpg.io/major-version` entry lives in the free-form
`Backup.spec.driverMetadata` map, which no downstream repository reads
by key.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backups, postgres): a Postgres restore whose backup was taken on another PostgreSQL major version than the target's spec.version now fails before touching the target, instead of purging it and then failing to recover.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Cross-major PostgreSQL restores are now refused before the target is
changed. The error identifies the backup and target versions.
* Restores proceed when either version is unavailable or cannot be
parsed.
* **New Features**
* Backup metadata now records the PostgreSQL major version when it is
available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 16:35:27 +03:00
Mattia Eleuteri 21fa1fcdb1 feat(postgres): add per-user createRole and bypassRls attributes
Without these, a tenant that needs to manage its own roles has no option
but the postgres superuser, which also hands over filesystem access and
command execution on the instance. Role attributes are not inherited
through role membership, so the generated <db>_admin group role cannot
carry them: they have to be set on the login user itself.

users.<name>.createRole and users.<name>.bypassRls set CREATEROLE and
BYPASSRLS on a managed login user. Both are reconciled on every
init-script run, so setting one back to false emits the NO... form and
revokes the attribute.

createRole is refused on PostgreSQL 15 and older. Before PostgreSQL 16,
CREATEROLE allowed granting membership in any non-superuser role, so a
user holding it could grant itself pg_execute_server_program and reach
command execution on the instance, which is exactly what not exposing
the superuser prevents. PostgreSQL 16 restricted CREATEROLE to roles the
grantee administers. bypassRls carries no such gate: BYPASSRLS only
waives row-level security inside the tenant's own databases, whose owner
already reads every row, so it adds no reach.

The two attributes are needed together, not separately. PostgreSQL only
lets a role create other roles carrying BYPASSRLS if it holds the
attribute itself, so provisioning a privileged service role fails with
"Only roles with the BYPASSRLS attribute may create roles with the
BYPASSRLS attribute" when only createRole is granted.

createRole also sets createrole_self_grant as a role-level default.
PostgreSQL 16 grants a CREATEROLE holder only ADMIN OPTION on the roles
it creates, governed by that parameter, whose default is empty. ADMIN
OPTION allows granting the role onward but not SET ROLE to it, and
creating an object owned by another role requires SET. The attribute
therefore looks functional -- CREATE ROLE succeeds -- while the next
statement a tenant writes, CREATE SCHEMA ... AUTHORIZATION <new role>,
fails with "must be able to SET ROLE". The default is reset when the
attribute is withdrawn.

The init job runs as the superuser, and before this change it trusted
two things a CREATEROLE holder controls for the roles it creates: role
comments and role names. It picked the users and roles to drop by their
comment, and a PostgreSQL 16 CREATEROLE holder may comment every role it
created, then hand it to the chart for deletion. It adopted any role
that already existed under a declared name, so a role created under a
name an administrator declares later stayed under its creator's control.
Neither can be caught from the catalog: creating a role through an
intermediate role and then dropping that role leaves the squatted role
with no member at all, while the objects, settings and default
privileges planted in it stay.

The chart therefore records the roles it manages as members of
"cozystack:managed", a role only the superuser can grant, and the delete
passes act only on those. Its first run marks every role the chart had
commented before, when no role could yet hold CREATEROLE through it.
Once any user has held createRole, recorded as a setting of the postgres
database, the chart refuses to adopt a role it did not mark and fails
with the role's name, rather than hand a squatted role REPLICATION or
the ownership of a database.

Verified against live PostgreSQL instances. On 17 a migration checklist
replayed as the login user, with no superuser: role creation with
attributes, CREATE SCHEMA ... AUTHORIZATION with the requested owner,
extensions in a named schema, per-role search_path and timeouts, RLS
including FORCE ROW LEVEL SECURITY, and SECURITY DEFINER functions all
succeed. On 17 the forged-comment and squatting paths are refused or
ignored, the upgrade from the previous chart marks the roles it already
managed, and a fresh install still adopts the role CNPG bootstraps for
users.app. On 13, 15 and 16 the init script runs and re-runs cleanly.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-06 16:26:59 +03:00
Aleksei Sviridkin 9734a16692 feat(postgres): add a postgis image flavor (#4587)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Adds a `flavor: postgresql | postgis` enum to the `postgres` app.
`postgis` runs the cluster on the CloudNativePG PostGIS operand image,
so tenants can enable `postgis`, `postgis_raster`, `postgis_topology`,
`postgis_sfcgal`, `postgis_tiger_geocoder` and `address_standardizer`
through the existing `databases.<name>.extensions`. The default stays
`postgresql` and renders exactly the image it rendered before this
change.

This picks up #2672 by @matthieu-robin, which the stale bot closed after
a rebase request, with these changes:

- The postgis tags in #2672 (`17.7-3.5`, `16.11-3.5`, ...) do not exist
in `ghcr.io/cloudnative-pg/postgis`; the registry answers 404, so the
flavor would have ended in `ImagePullBackOff`. The map now uses tags
that exist, `<minor>-<postgis>-standard-trixie`, on the same PostgreSQL
minor as `files/versions.yaml`.
- v18 is supported: PostGIS images for 18 now exist upstream.
- `standard` rather than `system`: the chart archives through the
barman-cloud plugin since the backup rework, so the operand no longer
needs the barman binaries of the deprecated `system` variant.
- `hack/update-versions.sh` derives the postgis map from the minors it
picks, and `hack/check-postgres-postgis-versions.bats` fails if the two
maps drift to different minors or away from `standard-trixie`.
- The chart refuses a flavor change on a live `Cluster`. The default
images are built on Debian bullseye (glibc 2.31) and the postgis ones on
trixie (glibc 2.41): CNPG would roll the swap out as a plain image
update, under an initialised data directory, which silently invalidates
every index on collatable text, and the postgis-to-postgresql direction
also drops the libraries behind every PostGIS object. Only the image
name is compared, so minor bumps and registry mirrors keep passing.
- `pgrouting` is not claimed: it is absent from the `18.1-3.6.1` build
the chart pins and only appears in later postgis builds.

A general mechanism (CNPG ImageVolume extensions) was considered and
left out: it needs PostgreSQL 18 and Kubernetes 1.35 or the
`ImageVolume` gate, so it cannot serve v13-v17 nor most management
clusters today, and PostGIS is the only extension CloudNativePG ships as
a full operand image. No other extension request exists in this
repository.

Testing:

- `helm unittest packages/apps/postgres`: 145/145, including 6 new cases
in `tests/flavor_test.yaml` (default image, postgis on the Cluster and
on the init Job, the guard in both directions with a mocked live
Cluster, and a mirrored digest-pinned image taking a new minor).
Removing the guard makes exactly the two guard cases fail.
- `hack/cozytest.sh hack/check-postgres-postgis-versions.bats` passes,
and fails on a shifted minor or a `system` tag.
- `make -C packages/apps/postgres generate` leaves no diff.
- The rendered
`ghcr.io/cloudnative-pg/postgis:18.1-3.6.1-standard-trixie` was started
locally: `CREATE EXTENSION` for citext, postgis, postgis_raster and
postgis_topology succeeds and `ST_Buffer` answers. A three-instance
cluster on the `system` sibling of that image has been running PostGIS
in production since 2026-09-29.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: cozystack/terraform-provider-cozystack#63 (issue: the
provider models every field by hand, and the default handling needs a
decision there)
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(postgres): add a `flavor` field accepting `postgresql` (default) or `postgis`. `postgis` bootstraps the cluster from the CloudNativePG PostGIS image, so PostGIS extensions can be enabled per database through `databases.<name>.extensions`. The flavor is fixed at creation: changing it on an existing cluster is refused, and so is restoring a backup into a cluster of the other flavor, whether through a RestoreJob or `bootstrap.enabled`; move data across flavors with a logical dump.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added a `flavor` option for PostgreSQL clusters. The default
`postgresql` flavor remains available, while `postgis` selects an image
with PostGIS extensions.
  - PostGIS images are available for PostgreSQL versions 13–18.
- Cluster flavor is fixed at creation. To switch flavors, create a new
cluster and migrate data with a logical dump.
- **Bug Fixes**
- Restores are refused when the backup and target cluster use different
flavors.
- **Documentation**
- Updated PostgreSQL configuration guidance to describe flavor options
and limitations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 16:17:22 +03:00
Mattia Eleuteri 39bdc9c404 fix(vm-instance): make the netplan network-data opt-in
Keying the network-data on instanceProfile reached every VM that never
set the field, because the profile defaults to ubuntu whatever image
the disk holds. The sysconfig, NetworkManager and eni renderers turn
each match stanza into an interface literally named after the netplan
ID, so a non-netplan guest left on the default profile would lose the
DHCP it gets today. A VM with neither cloudInit nor sshKeys would also
gain a cloud-init disk on upgrade, and KubeVirt's NoCloud instance-id
is new to that guest, so its next boot would regenerate host keys as a
first boot.

The configuration also shadows a static NIC set up through netplan:
netplan emits one networkd file per ID, networkd applies only the first
file in lexical order that matches a link, and 10-netplan-cozy-en sorts
before the file for enp2s0. The external-VLAN recipe claimed otherwise.

An explicit netplanDhcp value, off by default, leaves every existing
VM rendering exactly as before and lets the documentation say when not
to turn it on.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-06 16:11:33 +03:00
Mattia Eleuteri b5181b0b9b fix(vm-instance): supply cloud-init network-data to netplan guests
A netplan guest configures only the interface cloud-init's fallback
configuration covers, and keeps none of the search domains DHCP offers
it, so two things the platform already puts on the wire never reach it.
virt-launcher runs a DHCP server on every bridge-bound interface,
secondary ones included, yet a VM joined to a VPC subnet leaves that
interface inert for want of a client asking. The same server advertises
the launcher pod's search domains as option 119, which systemd-networkd
discards because UseDomains= defaults to no, and a cluster domain under
.local then falls through to the mDNS exclusion in resolved and answers
REFUSED.

The virtual-machine chart carried a network-data template for the first
of those, gated on systemDisk.image. Splitting that chart into vm-disk
and vm-instance removed the template along with the field it read, and
neither came back.

Matching en* and eth* rather than naming interfaces by index keeps the
configuration correct whatever the guest enumerates its NICs as, which
is where the previous template was fragile.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-06 16:10:29 +03:00
Aleksei Sviridkin 0e99ec012a docs(observability): add a distributed tracing guide (#3772)
## What this PR does

Adds `docs/observability/distributed-tracing.md`, the operator and
tenant guide for distributed tracing (epic #3761, design
cozystack/community `design-proposals/distributed-tracing`).

The guide covers the per-tenant backend and collector (#3762, #3763):
the in-namespace OTLP endpoints, the standard `OTEL_*` variables to
point an instrumented workload at them, sampling, redaction and tenancy,
and a per-engine table of what each managed engine can emit itself.
Verification against the engines (#3766) showed that none of them pushes
OTLP spans server-side, so the guide's answer for every engine is
client-side instrumentation, with the ClickHouse span table as the one
partial exception.

It also covers shared-central tracing (#3775, with the per-tenant write
rate in #4648): hosting the store in tenant-root, opting a tenant in,
the shared disk and the write rate with its limits (counted before
sampling, a 3 MiB request cap, the allowance a pod start lets through,
at most 10 collector replicas), the trust boundary with the collector's
write-only and Grafana's read-only logins, the lifecycle windows on
enabling and disabling, and the order to stop hosting in. Those details
used to live in the `tracingCentral` field description, which #3775 cut
to a few sentences because it ships as field help in the dashboard.

Both #3775 and #4648 are merged, and every figure and refusal the guide
quotes was checked against the charts on `main`.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The guide is a file under `docs/` in this repository and changes no
package, values key or API, so no trigger in the downstream map applies.

### Release note

```release-note
docs(observability): add a distributed tracing guide covering the per-tenant collector, client-side instrumentation per managed engine, and shared-central tracing in tenant-root.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Added a distributed-tracing guide covering tenant-local VictoriaTraces
storage, OTLP collectors, workload configuration, engine tracing
limitations, sampling, and optional attribute redaction.
* Documented shared-central tracing prerequisites, including tenant-root
hosting and tenant opt-in, plus write-rate, request-size, and replica
limits; credentials and network rules; tenant isolation and lifecycle
behavior; and conditions that stop hosting.
  * Clarified that cross-signal correlation is not configured.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 14:39:00 +03:00
Alexey Artamonov b3103af693 docs(observability): add a distributed tracing guide
Tracing reaches a workload only if it is instrumented and pointed at
the collector, and the managed engines push no OTLP spans of their
own, so operators and tenants need one place that says what the
platform provides and what is left to the application: the in-namespace
OTLP endpoints and the standard OTEL_* variables, sampling and
redaction, and what each engine can and cannot emit.

Shared-central tracing has more to it than the Tenant field description
can carry, since that text is shown as field help in the dashboard:
hosting the store in tenant-root, the shared disk and the per-tenant
write rate with its limits, the write-only and read-only logins and the
trust boundary, the windows on turning it on and off, and the order to
stop hosting in. The guide holds that detail.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 14:34:55 +03:00
Aleksei Sviridkin 11499c30e1 feat(monitoring): hold each shared-central tenant to a write rate (#4648)
## What this PR does

Closes the part of the distributed tracing design that #3775 left open:
the per-tenant collector rate-limits its tenant, so that a noisy or
hostile tenant cannot exhaust a shared backend (cozystack/community
`design-proposals/distributed-tracing`, Security and §2).

**Why the disk cap is not enough.** #3775 bounds every traces store by
disk usage, but VictoriaTraces always keeps the last two days, so a
tenant writing faster than the cap allows over two days fills the disk
every tenant on the shared store writes to. Only a per-tenant write rate
closes that.

**The collector is built here.** No upstream OpenTelemetry distribution
or processor limits throughput, and neither the open-source vmauth nor
VictoriaTraces limits a tenant's rate. Elastic's EDOT Collector
registers Elastic's `ratelimitprocessor` among its default components,
but it ships under the Elastic License 2.0 and carries far more
components than the chart uses. The collector is therefore built with
the OpenTelemetry Collector Builder from the upstream components the
chart already uses, at the same v0.136.0, plus that `ratelimitprocessor`
(Apache-2.0). Its v0.8.2 is the release built against collector
v0.135.0, the last one before v0.137.0, and it builds against v0.136.0
unchanged. The image is versioned by the collector release with a
cozystack respin counter (`v0.136.0-cozystack.0`), runs as the upstream
image's user, and is built with the other package images.

**How the rate is set and enforced.** The platform sets the rate with
`monitoring.tracingCentralTenantBytesPerSecond`, and it reaches every
tenant in `_cluster` through the `cozystack-values` Secret. A change
there is a change in the values every monitoring HelmRelease reads, so
Flux upgrades each of them; a rate read with `lookup` at render time
would never reach a tenant whose release is already in sync. The
platform writes the key only for a positive rate, because every release
in every tenant reads that Secret, and refuses to render on a value that
is not a whole number of bytes per second in plain decimal digits, up to
1 TiB/s ("1Mi", 1.5, "0100"), rather than reading it as the default.
Without it, or on a value that is not a positive number, the collector
holds the tenant to the default of about 97 KiB/s, at which one tenant
writes at most the disk cap of a default 10Gi store over two days,
counted uncompressed; the store keeps spans compressed, so they take
less than that. A larger store needs the rate raised with it. Each
central tenant's collector splits the rate across its replicas and
refuses data above it before the batch processor, so the application
sees the refusal, and right after the memory limiter, ahead of the
sampler, which adds a tracestate to every span it keeps, and of the
tenant attribute added to every resource, so it counts a request as the
receiver admitted it. The rate is therefore counted before sampling, and
the store keeps only the sampled share of it. Over OTLP/HTTP it gets a
429, which OTLP exporters retry. Over OTLP/gRPC the limiter answers
RESOURCE_EXHAUSTED without RetryInfo, which the OTLP specification makes
final, so a gRPC exporter drops those spans; the Tenant's
`tracingCentral`, the Monitoring `tracingCollector` and the platform key
tell central tenants to export over HTTP. The limiter sees the requests
the receiver admits, so in central mode the receiver caps a request at 3
MiB over both OTLP/HTTP and gRPC, decompressed for HTTP. The limiter
counts a request as the collector re-encodes it, which can be larger
than it arrived (13% for a span carrying only its IDs, the smallest
valid one, and up to 1.4x for spans with empty-valued attributes), so
burst is at least 5 MiB, above 1.4 times the cap, since a request larger
than burst never passes. Each collector pod starts with a full bucket,
so a pod start lets through above the rate up to 5 MiB or one second of
the pod's share of the rate, whichever is larger, and the chart refuses
to render a central tenant's collector with more than 10 replicas.

**Only the collector can write.** The rate holds only if the collector's
login is the only way the tenant writes. Each central tenant now has two
VMUsers on its one account: the collector's, which may only write, and
its Grafana datasource's, which may only read, each with its own Secret.
With one login for both, any user the tenant admits to its Grafana, a
Viewer included, could POST through the datasource proxy to `../insert`,
which vmauth cleans before it picks a route, and write into the shared
store past the limiter. The tenant's Grafana pod may reach the vmauth
too, so its image is the platform's, like the collector's, and only the
`auth.ini` key is mounted from the Secret a tenant names in
`oidc.customConfig.secretRef`: an image of the tenant's choosing with
the collector's Secret mounted would otherwise write past the limiter as
well.

**The collector image is the platform's.** The apps API defaults a
Monitoring spec from its schema but does not prune keys the schema
leaves out, and the Monitoring chart hands its values to the inner chart
as they are, so a tenant could set `tracingCollector.image` and run any
collector it liked, one without the limiter included. That key is no
longer passed through.

**Trade-offs.** The limiter is a vendor component, not an OpenTelemetry
one, and Elastic marks it as in development; it moves with Elastic's
releases, so bumping the collector means picking the matching release of
it (v0.74.0 builds against collector v1.67.0), and collector v0.136.0 is
a year old. It logs each refused request at error level, so a tenant
held at its limit produces steady log lines in its own collector.
Several tenants writing at their limits together can still exceed the
cap within two days; the rate bounds each tenant, not their sum. The
limiter keeps its state in each pod's memory, so a tenant admin who
restarts its collector pods over and over gains, per pod start, up to 5
MiB or one second of the pod's share of the rate, whichever is larger;
closing that needs a limiter with shared state (the processor's
gubernator mode), which is out of scope here. A sender whose connection
stays on one replica gets that replica's share of the rate. In central
mode the receiver refuses a request larger than 3 MiB: over OTLP/HTTP
with a 400, which senders do not retry, where the receiver default is 20
MiB, and over gRPC, where the default is 4 MiB. Changing the rate
upgrades every release in every tenant once, since it travels in the
Secret they all read. Until CI builds the image,
`tracingCollector.image` in `values.yaml` still names the upstream
distribution, which has no rate limiter, so a central-mode collector
from an unbuilt tree does not start; CI and releases stamp the built
image, and the image repository in the build registry has to be public
before a cluster can pull it.

**Upgrade.** Collectors in both modes move to the built image, which
carries the same components at the same version; central collectors also
roll for the new processor. The replica cap fails the render of a
central tenant's collector with more than 10 replicas, while a collector
writing to the tenant's own store keeps any replica count; the collector
has shipped only in the v1.7.0 alphas. Each central tenant's existing
VMUser loses its read route, and a second VMUser and Secret appear for
Grafana, whose datasource moves to the new login in the same release, so
the tenant's trace reads may fail until vmauth loads the new user.
Nothing else changes for clusters that do not set the rate: the platform
writes no new key, so no other release upgrades.

**Testing.** helm-unittest covers the platform writing the rate into
`_cluster` only for a positive whole number and refusing a malformed
one, numbers Helm reads as float64 included, the collector's limiter
(the configured rate, the split across replicas, burst, the receiver's
request cap in central mode and not in local mode, the default when
unset and on a word, zero or a negative value, its place right after the
memory limiter, with and without redaction, its absence in local mode),
the replica cap for a central tenant's collector and its absence for a
local one, the image key no longer passing through, the two logins (the
collector's with only the write route, Grafana's with only the read
route, separate Secrets and passwords), the Grafana image not passing
through, and only `auth.ini` being mounted from a `secretRef` Secret for
a central tenant; every term was checked by mutation. On a test cluster,
with the chart's rendered VMAuth, VMUsers, VTCluster, Grafana datasource
and collector: with one login for both, a Grafana Viewer's POST to
`../insert` through the datasource proxy landed a span in the shared
store; with the two logins vmauth answered `400 missing route` for every
encoding of the traversal tried, reads went through, and the collector's
login could write but not read. With the built collector at 100%
sampling and the receiver capped at 4 MiB, a fresh pod accepted a
request just under that cap made of 53-byte spans and one made of
ID-only spans, refused a larger one with a 400, and answered 429 once
the bucket was drained; the 53-byte spans it accepted were read back
through the read-only login; the 3 MiB cap the chart renders has not
been run on a cluster. A pod mounting a Secret through the chart's
`secretRef` volume as rendered for a central tenant got only `auth.ini`,
and one naming the collector's Secret, which has no such key, did not
start. The design's Testing section also asks for a sustained span flood
against a disk-capped store to show that the oldest spans are evicted
and ingest is not blocked; there is no end-to-end lane for
shared-central tracing yet, so that flood test is a follow-up, like the
cross-tenant isolation test #3775 names. On a test cluster, under the
network policies the tenant chart renders, the built image validated the
rendered config in both modes and wrote and read through the shared
store as before; a tenant given a small limit got 200 for its first
OTLP/HTTP requests and 429 for the rest of a burst, with the processor
logging each refusal, while another tenant under the default rate got
200 for all of the same requests.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#729
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The new platform key `monitoring.tracingCentralTenantBytesPerSecond`
belongs in the hand-written `platform-package.md` reference on the
website, tracked in cozystack/website#729. The Monitoring schema and the
Tenant's `tracingCentral` change only descriptions; the provider models
neither, so it needs nothing. The central-mode section #3772 adds to the
tracing guide covers the rate.

### Release note

```release-note
feat(monitoring): hold each shared-central tracing tenant to a write rate. The platform sets the rate with monitoring.tracingCentralTenantBytesPerSecond, and each central tenant's collector refuses spans above it, which bounds what one tenant writes into the store every tenant shares. The traces collector is now built by cozystack from the upstream components plus Elastic's rate-limit processor, and its image can no longer be set from a Monitoring application. A central tenant's Grafana reads the shared store with a login that cannot write, and its image can no longer be set from a Monitoring application.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Central tracing now applies a configurable per-tenant write-rate limit
shared across collector replicas. The default is approximately 97 KiB
per second, and collector replica counts are limited to 10.
* Each starting replica can temporarily exceed its rate share by up to 5
MiB, or by one second of that share when it is larger. Tenants share
central storage, so combined writes can fill the store and cause older
traces to be evicted.

* **Documentation**
* OTLP/HTTP is recommended for central tracing: excess traffic is
retried over HTTP, while OTLP/gRPC traffic may be dropped.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 13:29:16 +03:00
Alexey Artamonov e63b9e439c fix(monitoring): count a central request as the receiver admitted it
The limiter refuses for good any request it counts above burst, and
burst was the receiver's 4 MiB cap. A request grows on its way to the
limiter: the sampler adds a tracestate to every span it keeps, and the
limiter counts a request as the collector re-encodes it, which writes
the empty value of every attribute the wire may omit, so a valid
request of spans with empty-valued attributes is counted 1.4x. A
request the receiver admitted near its cap was therefore refused on
every retry.

The limiter now runs right after memory_limiter, so it counts the
request before sampling. The receiver caps a request at 3 MiB and
burst stays at 5 MiB, which holds 1.4x the cap; a larger burst would
also be what each pod start lets through above the rate. The rate is
counted before sampling, so the store keeps only the sampled share of
it. The texts say that a request above the cap gets a 400, which
senders do not retry, and that each collector rollout a tenant causes
lets that burst through again on every pod.

The replica bound that keeps that allowance small now applies only to
a central tenant's collector, whose render fails above ten replicas: a
collector writing to the tenant's own store has no rate, and the
replica counts it shipped with stay valid.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov 65f0caae2b fix(monitoring): keep a central tenant's write login out of its Grafana
A tenant's Grafana pod may reach the shared traces store's vmauth, and
the Secret a tenant names in oidc.customConfig.secretRef was mounted
into it whole. The apps API does not prune keys the Monitoring schema
leaves out, so a tenant could also set grafana.image: an image of its
choosing, with the collector's Secret mounted, writes past the rate
limit the collector enforces.

Grafana's image is the platform's, like the collector's, and for a
central tenant only the auth.ini key Grafana reads is mounted from the
named Secret. Other tenants have no collector login to keep out and
keep the whole Secret, which an auth.ini reading a sibling key with
$__file{} relies on.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00