Commit Graph
66 Commits
Author SHA1 Message Date
Myasnikov Daniil 87f79c17c3 fix(operator): keep scanning templates after the LimitRange settles
The steady-state fast path returned as soon as the LimitRange matched the
configured spec, on the grounds that LimitRanger gates anything a scan
could still find. That holds for pods and not for templates: a Deployment
added to a settled namespace with a memory request above the ceiling and
no limit of its own is admitted exactly as written, and only the pods it
then tries to create are rejected with "must be less than or equal to
memory limit". Nothing raised the ceiling, because every later reconcile
returned at the same line, so a chart bump became a rollout that could
never produce a pod and logged nothing about it.

Split the scan instead. Templates are read on every reconcile; the pod
list, which is the expensive half, is read only while no memory default
is in force in the namespace — once one is, LimitRanger fills in a limit
for every container admitted without one, so a container carrying a
request with no limit beside it can no longer come into existence. The
skip keys on the default actually being in force rather than on the
object existing, so a LimitRange whose spec was stripped scans pods again.

The write is now guarded by a semantic comparison against the spec the
scan settled on rather than against the configured spec, so a namespace
already sitting at a raised ceiling stops taking an identical apply on
every reconcile as well.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:46 +03:00
Myasnikov Daniil c9cacda1a0 docs(operator): pin the admission order by rule, not plugin index
The scan's justification for treating a VerticalPodAutoscaler as harmless
rested on LimitRanger being "position 6 of AllOrderedPlugins" and
MutatingAdmissionWebhook being "35". Both numbers are properties of one
release rather than of the ordering, and they had already drifted out
from under the comment. Under k8s 1.35, which this tree builds against,
LimitRanger is the 6th entry but MutatingAdmissionWebhook is the 33rd;
index 35 is ResourceQuota and the 35th entry is ValidatingAdmissionWebhook,
neither of which mutates, so the sentence named the wrong plugin under
either counting convention. In 1.31 the same plugin sat at 30, and
SecurityContextDeny being dropped in 1.35 is what moved LimitRanger.

Restate it on the invariant the list asserts about itself -- webhook,
resourcequota and deny plugins must go at the end -- with LimitRanger
compiled in above that marker and every plugin that can rewrite a
container's resources below it. That survives a release bump where an
index does not, and it covers MutatingAdmissionPolicy as well, a second
mutator that the positional wording predated and never accounted for.

Record what the order buys, which the old wording left implied. Run the
other way round, a plugin mutating before LimitRanger would write a
request into a container that has no limit yet, LimitRanger would then
default the limit to the configured ceiling, and core validation would
reject the fully mutated pod. The scan can see none of that -- the
request is in no template and the pod is never admitted -- so the order
is the only thing keeping VPA out of the residual gap this comment
already records for controlledValues: RequestsOnly.

Fix the matching rationale on the live-pod case in
TestFindRequestAboveDefaultLimit, which still claimed the pod's request
was raised by a webhook after the template was rendered. The ordering
rules that out; what the case actually rests on is a pod admitted before
the LimitRange existed, or one its controller builds from a custom
resource no template in the scan can show.

No behaviour change, and no new test: a container that carries its own
memory limit is already pinned out of the scan's scope for a template,
in TestFindRequestAboveDefaultLimit, and for a live pod, in
TestReconcileSystemDefaultsLimitRangeAppliesWhenLargeRequestCarriesItsOwnLimit.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:46 +03:00
Myasnikov Daniil 16b8ea4f0e docs(operator): correct the VPA rationale on the request scan
The comment argued that the danger this scan guards against is a VPA
webhook writing a request above the defaulted limit, which is then
rejected at admission. Admission ordering rules that out. LimitRanger is
position 6 of AllOrderedPlugins and MutatingAdmissionWebhook is 35, so
the LimitRange has already defaulted the container's memory limit before
VPA is called, and VPA rescales that limit in proportion to the request
it writes (GetProportionalLimit, under the default controlledValues:
RequestsAndLimits). Request and limit stay consistent and the pod is
admitted.

Rewrite it around what can actually happen: a request already in the spec
when it reaches admission with no limit beside it. In this tree that means
an operator lowering --system-namespace-memory-limit below a statically
declared request -- the largest is 2Gi, in
packages/system/rabbitmq-operator, so the 4Gi default clears them all
today. Two narrower shapes are real too: a pod that predates the
LimitRange, whose own controller may build it from a custom resource no
workload template in this scan can show; and a VPA set to
controlledValues: RequestsOnly, where nothing rescales the defaulted limit
and the rejection returns. The second doubles as the residual gap, since
a first pod rejected outright appears in neither a template nor a pod
list.

The scan itself is unchanged; only the justification was wrong.

Also record why withholding the LimitRange had to go rather than merely
being suboptimal: in a namespace with no LimitRange nothing defaults a
memory limit, so the next VPA-sized pod arrives with a large request and
no limit, the scan finds it again, and the namespace is withheld again.
The old behaviour could never get its foot in the door in cozy-monitoring.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:45 +03:00
Myasnikov Daniil d682158d1f test(operator): pin the tenant namespace boundary
One strings.HasPrefix keeps this feature out of tenant namespaces, and
nothing tested it: "tenant-" appeared twice in package_reconciler.go and
not once in the test file, so replacing the check with isSystem := true
left the whole package green.

What it protects is not subtle. packages/apps/tenant ships
tenant-range-limits, which defaults container memory to 128Mi in every
tenant namespace that declares resourceQuotas. A system LimitRange
landing beside it leaves two LimitRanges defaulting the same containers,
and which of them wins is decided by the order LimitRanger happens to
iterate them -- neither chart controls that -- so a tenant container's
default memory limit would move between 128Mi and 4Gi with nothing in the
tree changing.

Reconcile a variant with one system and one tenant namespace, and assert
no LimitRange is written for the tenant one. The system namespace in the
same run is the positive control: without it the test would still pass if
LimitRanges stopped being written anywhere at all. The cozystack.io/system
label is asserted too, since the same boundary gates both.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:44 +03:00
Myasnikov Daniil 2e1ab6682c fix(operator): raise the namespace ceiling instead of retracting it
When a container in a system namespace requests more memory than the
configured default, the guard used to withhold the LimitRange and delete
any it had applied earlier. That leaves the namespace with no default
memory limit at all, which is the precise state this feature exists to
remove: without memory.max on the pod cgroup the Talos OOM handler keeps
the pod in its victim set, and a tenant workload's memory pressure gets
paid for by a system component.

Default that namespace's memory to the offending request instead. A
looser ceiling still sets memory.max, so the pod scores zero in victim
selection exactly as it would at the configured limit -- 8Gi protects it
as well as 4Gi does, and both protect it infinitely better than nothing.
The raise needs no knowledge of where the request came from, and it
undoes itself: once the request is gone the scan finds nothing, desired
falls back to the configured limit, and the next reconcile applies it.

This is not a hypothetical namespace. packages/system/monitoring pins
maxAllowed.memory at 8Gi for vmselect and vmstorage,
packages/system/monitoring-agents pins 6G for vmagent, all three run
updateMode: Initial, and no monitoring template declares a memory limit.
Initial never evicts, so those requests do not come down on their own:
under the old behaviour cozy-monitoring, which holds the largest memory
consumers on the platform, was permanently excluded from the feature on
exactly the clusters it was written for.

A raised namespace no longer matches the base desired spec, so it scans
on every reconcile. That is deliberate: the raise has to track the
current largest request in order to be able to drop back, and caching it
would freeze a ceiling that nothing would ever lower again. The set of
raised namespaces is small.

The log moves from Error to Info. Nothing is wrong when this fires; the
operator is reporting a ceiling it chose.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:44 +03:00
Myasnikov Daniil c65db1ad52 perf(operator): skip the memory scan when the LimitRange is settled
reconcileSystemDefaultsLimitRange runs for every system namespace on
every Package reconcile, and Package reconciles follow HelmRelease status
churn through Owns, so on a platform with dozens of system namespaces the
pod and workload scan behind the guard is by far the most expensive thing
in the function -- and it is uncached by design, because caching it would
mean a cluster-wide Pod informer in an operator whose job here is to
bound memory.

Build the desired LimitRange first, read the existing one from the
informer cache, and return without scanning when the two specs already
match. That is safe because the LimitRange being applied is itself the
guarantee: from that moment LimitRanger rejects any new pod whose memory
request exceeds this default, so the only requests a scan could still
find are ones that predate the LimitRange, on pods already running, and
there is nothing it would do differently about them. Steady state becomes
one cached GET.

The comparison is apiequality.Semantic, not reflect.DeepEqual, because
resource.Quantity carries its parsed format and cached string: 4Gi and
4294967296 are the same ceiling and must not read as drift, or every
namespace would scan and re-apply forever.

The test counts List calls rather than only checking the resulting
LimitRange, which cannot tell a skipped scan from a repeated one that
reached the same answer, and seeds an 8Gi pod so a scan that ran would
also change the object.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:44 +03:00
Myasnikov Daniil b55c4f347a docs(operator): keep the residual-gap note on the request scan
The pod-only scan documented what it could not see: a request that only
ever exists post-admission, on a workload whose first pod is rejected
outright. Reading pod templates narrows that gap rather than closing it,
and the rewritten comment dropped the note entirely, which reads as a
claim that nothing is left.

Restore it in its narrower form: a template under the limit, a webhook
that raises the request past it, and a first pod rejected before it
exists, is still invisible to this scan.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:43 +03:00
Myasnikov Daniil 10acfe025d test(operator): cover both terminal pod phases in the skip
findRequestAboveDefaultLimit skips a pod in PodSucceeded or PodFailed,
because neither is ever recreated from that spec. Only the Succeeded half
had a fixture, so deleting the PodFailed clause left the whole ./internal
suite green -- half the condition was uncovered.

Give the case a Failed pod as well. Deleting either clause now fails it.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:43 +03:00
Myasnikov Daniil 654412acd2 fix(operator): scan workload templates for oversized memory requests
reconcileSystemDefaultsLimitRange withholds the LimitRange when a
container in the namespace requests more memory than the configured
default limit, because LimitRanger would then default a limit below that
request and the API server rejects the pod. The scan behind that decision
listed pods only, so it could not see a workload that currently has no
pods: a Deployment or StatefulSet at replicas: 0, a CronJob between runs,
a Job with spec.suspend. Those are ordinary steady states rather than
corner cases. The namespace looked safe, the LimitRange was applied, and
the pod was rejected at the next scale-up or the next schedule with "must
be less than or equal to memory limit of 4Gi" -- a workload that cannot
come back, discovered at the moment somebody needs it.

Scan the pod templates of Deployments, StatefulSets, DaemonSets, CronJobs
and Jobs as well as live pods. Both halves are needed: a template is the
only place a scaled-to-zero workload appears at all, and a live pod is
the only place a request that the VPA admission webhook raised after
admission appears. ReplicaSets are deliberately left out -- a live one
copies its Deployment's template, and the older revisions a Deployment
keeps carry superseded templates that only an explicit rollout undo would
ever instantiate, so scanning them would withhold the LimitRange over a
request replaced releases ago. Jobs are named separately because not
every Job has an owner whose template would cover it, and a finished one
is skipped for the same reason a terminated pod is. A suspended CronJob
is not skipped: suspend is the same dormancy as replicas: 0, undone by
the same kind of human action, and exempting it would leave the gap open
for one kind.

Templates are scanned before pods and the comparison stays strict, so a
workload and its own pods report once, as the workload -- the object an
operator has to edit, and one that outlives pod names. A live pod
outranks its own template only when its request is strictly larger, which
is exactly the webhook-injected case. memoryRequestBlocker therefore
carries Kind/name instead of a bare pod name, and the log key is
"workload" rather than "pod".

Every added List fails closed, which the caller already relies on: a scan
that cannot read is not evidence that applying is safe, so the namespace
is skipped and logged instead. The test for that seeds a sentinel
LimitRange, because asserting the object still exists cannot tell "left
alone" from "applied blind" -- a swallowed List error applies a fresh
LimitRange that exists just the same.

RBAC needs no chart change: the operator's ServiceAccount is bound to
cluster-admin, so it can already list these kinds. The kubebuilder
markers are extended anyway to record what the controller reads.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:42 +03:00
Myasnikov Daniil 7cc090b05f test(operator): pin the largest-offender scan to all three fields
The "largest offender is reported" case asserted only the container
name, and systemPod named every container "app", so the assertion held
whichever of the two pods findRequestAboveDefaultLimit had picked. The
comparison that keeps the largest request was uncovered.

Name the fixture container after its pod, assert the reported pod,
container and quantity together, and give the case three offenders whose
largest sits in the middle of the name-ordered pod listing, so keeping
the first, the last or the smallest offender all name a different pod
and quantity. The init-container case now pins the same three fields.

No production behaviour changes.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:42 +03:00
Myasnikov Daniil 546d8e6a54 feat(operator): default system namespace memory via LimitRange
The Talos userspace OOM handler (v1.12+) picks its victim by ranking
cgroups with `memory_max.hasValue() ? 0.0 : {Besteffort: 1.0, Burstable:
0.5, ...}[class] * memory_current`, discarding every cgroup that scores
zero. Only a memory limit sets memory.max on the pod cgroup, so only a
limit takes a pod out of the candidate set: a request merely demotes
BestEffort to Burstable, which under the v1.13.6 default
strictCgroupClassOrdering still means "killed second". PriorityClass has
no effect at all, because the handler walks cgroupfs and never talks to
the API server.

Victim selection is decoupled from the trigger, so the cgroup that
caused the pressure and the cgroup that gets SIGKILLed are unrelated by
design. Tenant workloads generally carry limits and system components
generally did not, which inverted the intended order: a tenant workload
thrashing against its own limit drives node-wide memory PSI, and
metallb-speaker or linstor-satellite is killed for it while the workload
responsible is never touched.

An audit found roughly 70 workloads across ~30 packages with at least
one container missing a memory limit, so per-chart resources would
neither cover the current set nor keep covering it. Maintain a
LimitRange in every system namespace instead, at the single choke point
that already creates them and already distinguishes system from tenant
(reconcileNamespaces). Tenant namespaces are left alone because
packages/apps/tenant owns LimitRange policy there.

The limit is a ceiling, not a reservation: it defaults well above real
usage (4Gi against a largest observed system request of 512Mi).
defaultRequest is set explicitly and small, because Kubernetes otherwise
defaults each request to the limit and reserves that much at schedule
time. Setting the limit to 0 disables the LimitRange and removes any the
operator previously created.

Two things keep the feature from being able to hurt more than it helps.

A LimitRange default only reaches a container that declares no memory
limit of its own, and the API server then rejects a request above its
limit. So before applying, scan the namespace for a container that
requests more than the configured limit and sets none, and where one
exists skip the namespace, retract any LimitRange applied earlier, and
log the pod, container, request and limit. This is not hypothetical:
packages/system/monitoring and packages/system/monitoring-agents ship
VPAs for cozy-monitoring with maxAllowed.memory of 8Gi and 6G under
updateMode: Initial, so the VPA admission webhook -- which runs after
the in-tree LimitRanger -- can write a request past 4Gi that no chart in
the tree declares. That is why the check runs against live pods rather
than rendered templates, and why it reads through APIReader rather than
starting a cluster-wide Pod informer.

And no failure inside the feature propagates: a LimitRange error, or a
pod list that cannot be read, is logged and stepped over instead of
failing the Package reconcile. A namespace without a LimitRange is back
to the behaviour of every earlier release, while a namespace that never
reconciles leaves its components uninstalled.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:41 +03:00
IvanHunters b0e07cadbb fix(operator): rebuild each changed ArtifactGenerator generation once (#4594)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

After an upgrade, an ArtifactGenerator can report Ready on its new spec
while its artifacts were built from the old one, and source-watcher
never rebuilds them. That is how the bucket-system chart in #4593
shipped without cozy-lib.

The cause is upstream (fluxcd/source-watcher#384). source-watcher builds
from the spec it read when the reconcile started, but the final status
patch stamps Ready with the live generation. An upgrade changes the
packages source and the copy rules within seconds, so a spec change
often lands during a build.

The generator status has nothing that tells a stale build from a good
one. So the operator now rebuilds every changed generation once, after
source-watcher first reports it Ready, through the force-drift path the
operator already uses for fluxcd/pkg#934. An unchanged spec rebuilds to
the same digest, so ExternalArtifacts and HelmReleases do not move.
Generation 1 is skipped, since the race needs a spec change during a
build.

On the first rollout every generator past generation 1 gets one extra
rebuild, and its PackageSource shows
`Unknown/AwaitingSourceWatcherRecovery` until that rebuild lands.
Nothing waits on that condition today. The same rebuild repairs clusters
already stuck like #4593.

The recovery messages on the PackageSource no longer blame
fluxcd/pkg#934, because the same path now serves both cases.

### How it was tested

Unit tests in `internal/operator/packagesource_reconciler_test.go` cover
the forced rebuild on generation 2 and on a later generation, every case
where no rebuild must happen, the rebuild settling without a second one,
both failure orders (force fails, recording the generation fails), and a
stale recovery counter. Each condition of the new predicate was mutated
once to confirm a test fails.

I also ran this commit's operator image on an arm64 test cluster running
v1.7.0-alpha.3, the one where #4593 was found. Before that I had
repaired its bucket generator by hand: deleting the stale
ExternalArtifact and requesting a reconcile produced an artifact with
`charts/cozy-lib` and both bucket HelmReleases went Ready. With the
patched operator, the only generator past generation 1 got exactly one
forced rebuild and the verified-generation annotation. source-watcher
reported all its ExternalArtifacts up to date. The digests of all 214
ExternalArtifacts and the revisions of all 104 HelmReleases matched a
snapshot taken before the swap, and every PackageSource and HelmRelease
stayed Ready. The race itself was not reproduced on the cluster, since
that needs a spec change landing during a build.

Closes #4593

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(operator): rebuild an ArtifactGenerator once after each spec change, so an upgrade no longer leaves a component on artifacts built from the previous spec (for example a chart missing a library it now needs).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved PackageSource recovery when an artifact generator reports
ready: the system now verifies the generated output and triggers a
rebuild when needed.
* Failed rebuild or verification attempts can be retried during a later
reconciliation, and recovery tracking is cleared once the rebuilt
generation settles.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:43:44 +03:00
Aleksei Sviridkin adc2dcb5ca fix(operator): rebuild each changed ArtifactGenerator generation once
source-watcher builds an ArtifactGenerator from the spec it read when
its reconcile started, but its final status patch re-reads the live
object and marks Ready with the live generation. If the spec changes
during a build, which an upgrade does as it rolls out a new packages
source and new copy rules within seconds of each other, the generator
reports the new generation as built while its artifacts come from the
old one, and source-watcher never rebuilds them. A chart can then ship
without a library its templates now require.

The generator status cannot tell a stale build from a good one, so the
operator forces one rebuild of every changed generation once it
reports Ready, through the existing force-drift path. An unchanged spec
rebuilds to the same digest, so nothing downstream moves. The recovery
messages no longer name one upstream race, since they now serve two.

Upstream: fluxcd/source-watcher#384.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-30 01:22:49 +03:00
Aleksei Sviridkin a09eded9ec fix(kubernetes-worker-image): wait for goldens to import before Ready
The Talos worker image catalog renders its goldens the same way the VM
image collection does: DataVolumes with immediate bind whose import
starts at install. With no health check the release went Ready as soon
as helm applied them, so a golden that never imported left the catalog
green.

Gate it on the DataVolume phase with the same expressions. Worker
pools read each golden directly and do not wait on the catalog status,
so the gate changes what the status reports and delays no pool.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 22:15:59 +03:00
Aleksei Sviridkin 51ac18776e fix(vm-default-images): wait for golden images to import before Ready
Every image is a DataVolume with immediate bind, so its import starts at
install, but nothing waited for it: the HelmRelease went Ready as soon as
helm applied the DataVolumes. An image that failed or stalled left the
component green, and the failure surfaced later on a vm-disk cloning
from it, far from the cause.

Gate the release on the DataVolume phase. Only Succeeded counts as
ready, and a Failed phase, when CDI sets one, ends the wait at once.
Every other phase, and a missing status, keeps the wait open until the
install window expires. There is no per-component timeout, so a download
longer than that window is retried as an upgrade until it finishes, and
the retry strategy never uninstalls the volumes.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
2026-09-29 22:15:58 +03:00
Aleksei Sviridkin 7a0aab2bdc refactor: apply the Go 1.27 modernizers
Run go fix over every first-party module: interface{} becomes any,
hand-written loops become slices, maps, min/max and range-over-int,
errors.As becomes errors.AsType, pointer helpers become new(expr) and
are inlined at their call sites, and so on. Every rewrite keeps
behaviour.

Three rewrites stay out. embedlit flattens embedded-field literals
such as ObjectMeta, syntax only Go 1.27 accepts, which would break
backports to release branches still on Go 1.26. omitzero drops
omitempty from struct fields, and openapi-gen reads omitempty as
optional, so metadata, spec and status would turn required in the
published schemas. And errorsastype rewrote an errors.As whose
target, apierrors.APIStatus, does not implement error, which
errors.AsType rejects at compile time.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 17:23:18 +03:00
Aleksei Sviridkin 8f0f29e1bd test(operator): pin how component values files are layered
The first values file overwrites the component's values.yaml and each
later one merges into it; no test held that order or the strategies.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:55:03 +03:00
IvanHunters aa20495df3 feat(marketplace): register a tapped repository's apps on connect
Connecting a marketplace repository materialized its PackageSource but created no Package, so its ApplicationDefinitions never rendered and its apps never appeared in the dashboard catalog until a manual `cozypkg add` the dashboard cannot drive (the package list is computed from the ApplicationDefinitions, which do not exist yet).

The tap materializer now creates a tap-managed registration Package for each materialized PackageSource, the way the platform ships a Package per built-in app, so the ApplicationDefinition registrations deploy on connect and the apps become browsable; the app components carry no install block and are still instantiated per user resource.

Privileged install components of an unconfirmed tap auto-registration (a Package that still carries the marketplace-tap label) are refused on several best-effort layers: the Package reconciler skips their HelmRelease and prunes a lingering one at the install site, and the PackageSource reconciler withholds their ExternalArtifact so the helm-controller has no privileged content to deploy, re-running on confirmation via a Package watch. `cozypkg add --allow-privileged` sheds the label as an explicit ownership handover, which is the confirmation; a platform Package never carries the label. A benign component flipped to privileged in a new revision can still be materialized by source-watcher before these layers observe the flip; closing that race fully requires decoupling the catalog registration from install and is tracked in issue #4359.

`cozypkg untap` and the dashboard disconnect remove the registration Package (de-registering the apps) under a UID + ResourceVersion precondition, so a concurrent `cozypkg add` handover is never clobbered; running instances are left but unmanaged. `cozypkg add` re-opens a tap auto-registration for variant selection and hands it over. The cozystack-api ClusterRole gains delete on packages for the disconnect path.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-21 21:57:04 +03:00
scooby87 afc003c464 feat(postgres): database read-replica autoscaling via KEDA (#3954)
## What this PR does

Adds automatic horizontal autoscaling of a PostgreSQL database's **read
replicas** in response to load, implementing the merged design proposal
[Database Horizontal
Autoscaler](https://github.com/cozystack/community/tree/main/design-proposals/database-horizontal-autoscaling)
(community#44). The mechanism is entirely stock — no bespoke operator,
no new CRD:

- **KEDA as an optional platform component** (`packages/system/keda`,
wired into `core/platform` as a source + system-bundle entry). It claims
the `external.metrics.k8s.io` APIService that nothing served before.
- **`cozy-lib.keda.scaledObject` helper**
(`packages/library/cozy-lib/templates/_keda.tpl`) renders a KEDA
`ScaledObject`, fail-closed on missing/invalid arguments.
- **`packages/apps/postgres` autoscaling** — an `autoscaling` values
block (`enabled`, `minReplicas`, `maxReplicas`, `metric`, `target`,
`maxReplicationLagSeconds`, `dryRun`, `transition`) validated by
`values.schema.json`; under active autoscaling the chart renders a
`ScaledObject` and seeds `spec.instances` to a constant floor
(`max(replicas, effectiveMin)`), while the postgres release is forced to
client-side (two-way) Helm apply so the seed is a merge no-op on upgrade
and the KEDA-managed HPA — the runtime writer via the CNPG `Cluster`
scale subresource — is not reverted.
- **Single-value metric (§1)** — one PromQL query returns `Σ(read load
over standbys) + target`, so a stock HPA computes `desired =
ceil((Σ+target)/target) = 1 + ceil(Σ/target) = primary + desiredRead`,
with no per-pod emission.
- **The four static brakes (§5)** — quorum floor `max(minReplicas,
maxSyncReplicas+1, 2)` (quorum wins over `maxReplicas`), scale-down
pacing (`Pods/1/600s`), a replication-lag clamp folded into the query
(freeze both directions while lag exceeds the threshold and the primary
is writing, with `max_over_time` hysteresis), and dry-run (render
paused).
- **Two-phase enable/disable migration** via a `transition` sub-flag:
phase 1 stages `spec.instances` to the live count and brings KEDA up
with its HPA paused in both directions; flipping to phase 2 hands the
runtime count to KEDA, and because the seed re-renders the same count
the flip is a merge no-op with no dip.
- **Observability** — a database-autoscaling Grafana dashboard and
PrometheusRule alerts.

**Live validation (dev cluster, CNPG 1.30.0).** The full path works
end-to-end: the CNPG `Cluster` scale subresource serves
`status.selector`, KEDA renders the HPA, the metric reaches it
(`ScalingActive=ValidMetricFound`, `TARGETS 151/150`, `desiredReplicas=2
= 1 primary + 1 replica` — the §1 arithmetic), and the lag-clamp query
returns the base value on live VictoriaMetrics. Two packaging fixes were
needed and are included: KEDA must run on the cozystack `cozy.local`
cluster domain (the upstream `cluster.local` default makes the metrics
gRPC target NXDOMAIN and every HPA hangs on `FailedGetExternalMetric`),
and the operator's client-go throttle is raised so a single-replica
operator does not lose its leader lease under scale-subresource polling.

Implements #3953.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. Two repositories are affected; both are follow-ups to open once
this lands (and after its dependency #3951 merges and the values surface
is final), left unticked here for a maintainer to confirm rather than
claimed with a speculative link:

- **cozystack/terraform-provider-cozystack** — this adds the nested
`autoscaling` block to `packages/apps/postgres/values.schema.json`. The
provider hand-models each app's fields (schema + model +
expand/flatten), so it needs the new block to expose it in Terraform.
- **cozystack/website** — KEDA is a new platform component
(`guides/platform-stack`), and the postgres reference page gains the
`autoscaling` params. The managed-app reference is regenerated from
`README.md` by the release bot; the platform-stack component list is
hand-maintained.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: to open (new platform component + postgres autoscaling
params)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: to open (new `autoscaling` block in postgres
values.schema.json)

### Depends on

- #3951 — bump CloudNativePG to 1.30.0. The HPA can only drive the CNPG
`Cluster` scale subresource on CNPG ≥ 1.28.4, where it serves
`status.selector`; without it the HPA fails with `InvalidSelector`. This
PR should not merge before #3951.

### Release note

```release-note
feat(postgres): add optional horizontal autoscaling of read replicas via KEDA, configured through the new `autoscaling` values block; KEDA ships as a new optional platform component. Requires CloudNativePG >= 1.28.4 for the Cluster scale subresource selector. Note: every Postgres release now applies client-side (`release.cozystack.io/helm-server-side-apply: "false"`) so the autoscaler's live `/scale` count is not reverted by Helm; a kind-wide side effect is that chart-rendered fields on Postgres releases are no longer re-asserted by server-side-apply drift correction (driftDetection is unset), a documented trade-off with a per-instance apply strategy as follow-up.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added configurable PostgreSQL read-replica autoscaling with KEDA.
* Added CPU and read-connection metrics, replica bounds, replication-lag
safeguards, dry-run mode, and transition handling.
* Added optional KEDA platform support, Grafana autoscaling dashboard,
and Prometheus alerts.

* **Documentation**
* Clarified integer thresholds, prerequisites, configuration, and
transition guidance.

* **Bug Fixes**
* Improved validation and actionable errors for unavailable KEDA or
invalid settings.

* **Tests**
* Expanded coverage for autoscaling, observability, package selection,
and validation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-09 21:28:34 +03:00
IvanHunters 8738063287 fix(operator): reject tar entries containing ".." before any path operation
safeJoin already rejected traversal and re-checked the joined path against
destDir, but the guard sits behind a call boundary that CodeQL's go/zipslip
barrier does not follow, so the extraction loop reported three high-severity
arbitrary-write alerts. Reject the entry name up front instead, which is both
the fail-closed spelling and the shape the analysis recognises. Extend the
traversal test to cover a ".." buried mid-path, not just a leading one.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-07 10:58:40 +03:00
IvanHunters 1653da967b refactor(marketplace): drop artifact-prefix injection and reserved names
Without the community. rename, a tapped repository's cozyrds
ApplicationDefinition references its component's artifact by the concrete
<packagesource>-<variant>-<component> name again, so the operator no longer
injects a cozystackArtifactPrefix value and the -rd chart renders its cozyrds
asset with a plain Files.Get. cozypkg init scaffolds the concrete chartRef.

Reserved name prefixes are removed: repositories choose their own PackageSource
names, and a clash with a core component is caught on-cluster at tap time (the
collision check) rather than by refusing prefixes offline.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-07 10:58:38 +03:00
IvanHunters 95badea1c8 refactor(marketplace): identify taps by marker label, drop community rename
Tapped repositories keep their own declared PackageSource names instead of
being renamed under the community. prefix. Tap-ness is now marked by the
marketplace-tap label (and source annotation) on the OCIRepository and every
materialized PackageSource, so untap, the Flux-source cleanup, and idempotent
re-tap key on the marker rather than a name prefix.

A declared name that clashes with an existing core component (or another tap)
is rejected: the CLI tap fails fast before applying anything, and the operator
materializer refuses to overwrite the foreign object, records the reason on the
source (annotation and Event), and does not stamp the revision so a corrected
artifact retries. The shared check lives in internal/marketplace/collision.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-07 10:58:38 +03:00
IvanHunters 0c0609ad63 fix(marketplace): resolve community chartRef via injected artifact prefix
Fixes the rename-vs-chartRef break found on dev3: a community-tapped
PackageSource is renamed to community.<org>.<repo>, so the assembled artifact
name changes, but a repo's cozyrds ApplicationDefinition baked chartRef.name
under the author's original name and the ExternalArtifact it referenced no
longer existed, leaving every community-installed app permanently ArtifactFailed.

Implements @kvaps's option 2 (derive at the -rd component, keep platform charts
literal):
- naming.ArtifactPrefix exposes the per-(source,variant) artifact-name prefix.
- The Package reconciler injects it as the cozystackArtifactPrefix Helm value,
  but ONLY for community.-prefixed sources, so platform HelmRelease values are
  byte-for-byte unchanged (no churn, no platform chart edits).
- cozypkg init now scaffolds a community-style -rd chart: cozyrd.yaml renders the
  cozyrds asset through tpl, and chartRef.name templates
  '{{ .Values.cozystackArtifactPrefix }}-<app>', resolving to the real artifact
  whatever the tap-time rename produces.
- validate skips the dangling-chartRef check for a templated name (it cannot be
  cross-checked at authoring time; the operator resolves it at install).

Verified end-to-end: init -> validate is clean, and helm template with the
prefix value renders chartRef.name to the exact artifact the reconciler emits.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-07 10:58:38 +03:00
IvanHunters 950268cccc feat(marketplace): reject reserved PackageSource name prefixes
From @kvaps's review: a repository author could name a PackageSource
'cozystack.<x>' or 'community.<x>' and claim the platform or marketplace
namespace. Until now validate only rejected an empty metadata.name and the
forced rename in the materializer was the sole guard.

- validate now flags 'cozystack.'- and 'community.'-prefixed PackageSource names
  as an error. The escape hatch --allow-reserved-names lives on the caller side
  only: the index gate (validate-entry.sh) never passes it, so a submitter
  cannot hand it to themselves.
- cozypkg init refuses the same prefixes early with a neutral-name hint.
- Document why the ClusterIP fail-open is safe: it falls back only on address
  resolution, and integrity stays enforced by the fail-closed digest check, so
  a wrong host yields a mismatch error rather than a content substitution.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-07 10:58:38 +03:00
IvanHunters b96a97e5c4 fix(marketplace): resolve tap artifact host to ClusterIP for the hostNetwork operator
The tap materializer fetches the Flux artifact from status.artifact.url, which is
a cluster-DNS name (flux.<ns>.svc/...). cozystack-operator runs with
hostNetwork: true and dnsPolicy: ClusterFirst, so it resolves against the host's
DNS rather than CoreDNS and that name is NXDOMAIN — every dashboard-driven
connect (API Create, which relies entirely on the materializer) then hangs in
'connecting' forever, while the CLI tap works only because it pulls via
'flux pull artifact' from the public registry.

Resolve the artifact host to the backing Service's ClusterIP (which a
hostNetwork pod reaches directly, the same way the in-cluster apiserver is
reached via KUBERNETES_SERVICE_HOST) before fetching. This fails open: a
non-cluster host, a missing Service, or a headless/empty ClusterIP falls back to
the original URL, so clusters where DNS already works are unaffected. Services
are readable through the operator's existing cluster-wide get/list/watch grant.

Found on dev3 via cozystack-pr-test; the connect->materialize path never worked
on a stock operator deployment before this.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-07 10:58:38 +03:00
IvanHunters c9accd6086 fix(marketplace): address maintainer review — url collisions, fail-closed digest, centralized prefix
From @kvaps's review of the untrusted-artifact path:
- Refuse to silently retarget: two repositories that share an org/repo path on
  different registry hosts derive the same tap name. The API and CLI now reject
  a connect when an existing source points at a different spec.url, instead of
  blindly repointing the first at the second.
- Fail closed on integrity: verifyAndExtract now errors on an absent digest
  (not just a mismatch), and the reconciler waits for status.artifact.digest,
  so a tap artifact is never materialized without a verified digest.
- Centralize the load-bearing community. prefix in internal/marketplace/tapconst
  (removing two duplicate consts and several literals across the CLI, API,
  materializer, and dashboard) so the disconnect guard, the community badge, and
  the anti-shadowing guarantee cannot drift.
- Split multi-document YAML with the apimachinery reader so a PackageSource
  whose values contain a separator-looking line is not fractured.

(The short-name version-pin drop was already fixed in an earlier commit.)

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-07 10:58:37 +03:00
IvanHunters 3ce8279019 fix(marketplace): prune stale taps, preserve source state, scope RBAC, pull by digest
Address the remaining review findings:
- The materializer now prunes PackageSources it materialized from an earlier
  revision that the current artifact no longer contains (including a single/multi
  rename), so a removed package leaves the catalog; installed Packages stay.
- A repeat connect updates the fetched OCIRepository's spec, label and name
  annotation instead of replacing the object, so the operator's finalizer and
  materialized-revision annotation survive; covered by a test.
- OCIRepository write verbs move from the cluster-wide cozystack-api ClusterRole
  to a namespaced Role in cozy-system (get/list/watch stay cluster-wide for the
  watch cache), since the API only manages tap sources there.
- 'cozypkg validate --require-signature' now pulls the exact manifest digest
  cosign verified, closing the tag-mutation window between verify and pull.
- safeJoin rejects any '..' path segment outright (defense in depth and to
  satisfy the Zip-Slip scanner) on top of the existing containment check.
- Document why the index gate's merge/label steps need a workflow_run split for
  fork pull requests.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-07 10:58:37 +03:00
IvanHunters 7b703f2859 fix(cozypkg): satisfy golangci-lint and derive a valid kind for hyphenated apps
Address the linters flagged in review: check the previously-ignored writer,
Close, and RemoveAll return values (errcheck), lowercase an error string
(ST1005), drop the embedded-field selector (QF1008), remove an unused const,
and switch the materializer's post-finalizer requeue from the deprecated
Result.Requeue to RequeueAfter.

Also fix 'cozypkg init': capitalize now builds an upper CamelCase kind from a
hyphenated --app (foo-bar -> FooBar) so the scaffolded ApplicationDefinition is
not rejected by the apiserver at install time; covered by a test.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-07 10:58:37 +03:00
IvanHunters 3643fceceb refactor(marketplace): fail-safe re-tap rollback and single-source artifact naming
Close the two non-blocking notes from the third review round:

- objectExists now fails safe: it returns 'not existing' (eligible for
  rollback) only on a definite NotFound; any other Get error (RBAC, throttling,
  timeout) is treated as possibly-pre-existing, so an idempotent re-tap that
  hits a transient probe error cannot delete a pre-existing resource on
  rollback. Mirrors the fail-direction of sourceStillReferenced; covered by a
  new interceptor-based test.
- The PackageSource reconciler now calls internal/marketplace/naming.ArtifactName
  instead of inlining the formula, so the artifact-name convention has a single
  definition shared by the producer, the marketplace backend, and the
  validator, and the parity test guards producer-vs-helper.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-07 10:58:37 +03:00
IvanHunters e61d2c7b53 feat(marketplace): connect a community tap from the API with async materialization
Add server-side connect to complete the marketplace write path. Creating a Tap
records the intent only: the API validates the oci:// reference and creates a
labeled Flux OCIRepository (with an optional pull-credential secretRef), never
blocking the request on a registry pull, and returns a Tap whose status shows
materialization pending. The connect inputs (url, tag, secretRef) are write-only
and are never echoed back or returned on read.

A new tap materializer reconciler in the operator watches community-tap
OCIRepositories, and once source-controller has pulled the artifact, fetches the
tarball over the Flux artifact URL (digest-verified, size-bounded, path-traversal
guarded), parses the PackageSource(s) it carries, and materializes them under the
community. prefix with their sourceRef repointed at the tap's source. A finalizer
cleans up the materialized PackageSources when the source is removed, leaving
installed Packages in place.

The shared label/annotation keys live in internal/marketplace/tapconst so the API
and the operator cannot drift on the connect-to-materialize handoff.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-07 10:58:35 +03:00
Alexey Artamonov 27d7dbb140 feat(api): add per-app helm apply-strategy override annotation
Add release.cozystack.io/helm-server-side-apply on the ApplicationDefinition,
threaded through ReleaseConfig into every emitted HelmRelease's
Install/Upgrade.ServerSideApply. Empty leaves the helm-controller default
(server-side apply on v1.5.0); "false"/"disabled" forces client-side apply.

The postgres read-replica autoscaler needs this: KEDA's HPA writes the CNPG
Cluster's /scale subresource (spec.instances) while the chart renders a constant
seed. Under helm-controller's default server-side apply the rendered field is
force-owned and reverts KEDA's live value on every reconcile. Under client-side
apply helm-controller patches the CRD from the previous-vs-new rendered manifest
without consulting live state, so the unchanged constant yields no patch and the
/scale value survives (verified against Helm v4 createPatch: unstructured/CRD
objects take the two-way jsonpatch.CreateMergePatch(original, modified) path
unless threeWayMergeForUnstructured is set, which helm-controller v1.5.0 leaves
at its default).

Guarded by TestConvertApplicationToHelmRelease_ServerSideApplyOverride on the
tenant-release builder (convertApplicationToHelmRelease), and by
TestParseHelmServerSideApplyAnnotation. Removes a stale package_reconciler_test
guard that asserted a false invariant on the wrong (platform Package) builder.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:19:25 +03:00
Alexey Artamonov 826f08972f fix(apps/postgres): make autoscaler fail-safe, dip-free, and observable
Address review findings on the KEDA read-replica autoscaler where the
implementation diverged from the merged design proposal.

Fail-safe read load. Gate the `or vector(0)` floor on the existence of ANY CNPG
instance pod (`count(kube_pod_labels{...instance_role=~".+"}) > 0`) — primary or
replica — so the query is empty ONLY when no CNPG pod labels exist at all (a
broken kube-state-metrics pipeline), not during initial provisioning when the
sole replica-side pod is CNPG's join Job (jobRole, not instanceRole). Also pin
`ignoreNullValues: "false"` on the trigger: KEDA's prometheus scaler defaults it
true, turning an EMPTY result into metric 0 (not an error) — the HPA would then
desire 0 and clamp to the floor, i.e. silently scale a broken-pipeline cluster
DOWN. With it false an empty result is a scaler error, so the HPA HOLDS and
DatabaseAutoscalerScalerErrors fires; gating on any instance role keeps a healthy
cluster (including a long basebackup) from tripping that alert.

Constant seed = max(replicas, effectiveMin), so enable/dryRun/transition flips
are a merge no-op with no dip and never undercut the quorum floor.

Fail closed when the quorum floor exceeds maxReplicas (was: effectiveMax silently
raised both bounds, unbounded for a large maxSyncReplicas).

Dry-run / transition keep the HPA (unidirectional pause pair, not the HPA-deleting
full pause) but both SelectPolicy=Disabled, so desired normalizes to current — the
recommendation is read from the read-load metric, not a diverging desired count.

Observability. DatabaseAutoscalerAtMax keys on desired >= maxReplicas, NOT the
ScalingLimited condition (which is also true at the floor, so it fired on every
idle cluster with wrong "raise maxReplicas" advice). Dashboard read-load panels
are per-cluster (label_cnpg_io_cluster) with a CPU panel for ReadCPUUtilization
and the exact clamp in descriptions; ScaleStuck's Pending intersection is scoped
to CNPG pods; alerts.yaml narrative comments moved to Go-template comments so they
do not ship into the PrometheusRule.

Guard the seeding invariant: TestBuildHelmReleaseSpecNoForceOrDriftDetection.
Regenerated README/schema/types.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:19:25 +03:00
Myasnikov Daniil 96a9652fb1 feat(api): typed waitStrategy/healthCheckExprs for kstatus readiness
Add first-class fields so a package/app can declare the Flux helm-controller
v1.5.0 kstatus health gate for the custom resource it wraps, so
HelmRelease.Ready means "the CR is actually healthy" instead of "helm applied
the CR" (#2642). API-owner approved in #3263.

- api: WaitStrategy (scalar, maps to HelmReleaseSpec.WaitStrategy.Name — a
  deliberate simplification) + HealthCheckExprs ([]kustomize.CustomHealthCheck,
  the upstream type verbatim) on both ApplicationDefinition.spec.release (the
  apiserver path) and ComponentInstall (the operator path).
- config.ResolveWaitStrategy: shared by both HelmRelease builders. Couples the
  default — healthCheckExprs imply the poller strategy when none is set, since
  exprs are only evaluated under poller. This removes the "must also set
  waitStrategy: poller" footgun and does not depend on the controller's global
  default.
- thread through buildHelmReleaseSpec (operator) and
  convertApplicationToHelmRelease (apiserver); the apiserver loads the typed
  spec.release fields in start.go.
- regenerate deepcopy, the ApplicationDefinition and PackageSource CRDs, and
  openapi. The backupstrategy CRDs also regenerate — benign k8s-0.35 PodSpec
  description drift, not a functional change.

disableWait is intentionally not added (YAGNI): the only known need (the
kubernetes tenant-addon deadlock) was resolved child-side, and waitStrategy:
legacy is not equivalent (legacy still waits Helm-v3-style; disableWait skips
waiting entirely). Add it later when a concrete leaf needs a true no-gate
escape hatch.

Refs #2642

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-16 16:45:16 +05:00
Ivan Okhotnikov 0d98e27981 fix(operator): hold final backoff window before give-up, note upstream version skew
Address lexfrei's non-blocking follow-ups on the LGTM approval:

Follow-up #1 — give-up preempted the final backoff. `decideRecovery`
returned `recoveryActionGiveUp` as soon as `attempts >= maxRecoveryAttempts`,
without checking whether the final-force backoff window had elapsed. The
Owns()-watch re-fire from the N-th force-drift's own writes triggered a
reconcile milliseconds after the force, so the effective time-to-give-up
was ~7.5m (30s+60s+2m+4m) — not the ~11.5m documented on the constants
block. The final 4-minute response window the last nudge was supposed to
grant source-watcher was never actually granted.

Fix: extend the exhaustion branch to hold `Wait` until
`now - lastRecoveryAt >= backoffFor(maxRecoveryAttempts)` before switching
to GiveUp. Now the documented ~11.5m budget matches the observed
behaviour, and the last force gets a real chance to succeed (Owns
watch → source-watcher Ready=True copy-through) before we surface
SourceWatcherStalled.

Tests:
* `TestDecideRecovery` gains a case pinning `Wait` when
  attempts=maxRecoveryAttempts but the final backoff has not yet
  elapsed (the exact re-fire shape the previous behaviour tripped on).
* `TestUpdateStatus_OwnMarkerAttemptsExhaustedButFinalBackoffPending`
  drives the full updateStatus path against the same shape, asserting
  the PackageSource stays in Unknown/AwaitingSourceWatcherRecovery and
  the RequeueAfter equals the remaining backoff, NOT zero (GiveUp).

Follow-up #3 — upstream version-skew note. The Go module dependency
tracks `fluxcd/source-watcher/api/v2 v2.0.3` (CRD types package only),
but the deployed binary this workaround targets is pinned to v2.1.0
at `internal/fluxinstall/manifests/fluxcd.yaml`. Comments referencing
`drift.go:52` / `controller.go:164/254` describe the deployed binary,
not the compiled API package. Documented on the constants block so a
future flux bump re-verifies the anchors AND the watch predicate /
drift-detection heuristics, which are the load-bearing upstream
assumptions.

Follow-up #2 (envtest over full Reconcile→Owns→updateStatus cycle) is
architectural and out of scope for this PR — noted for a follow-up.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
2026-07-05 08:30:19 +03:00
Ivan Okhotnikov 37421d2fec docs(operator): honest JSON-merge-patch tradeoff, clarify asymmetric OG check, warn on partial force-drift
Self-review nits raised in a skeptical pass over the current tip:

* The status-patch docstring in forceArtifactGeneratorDrift described
  the JSON-merge-patch full-array-replace on `status.conditions[]` as
  "net-neutral". That was too optimistic: if source-watcher writes a
  concurrent Reconciling condition between our Get and our Patch, our
  patch does transiently wipe it. source-watcher restores it on its
  next reconcile (which our requestedAt bump is about to trigger), but
  downstream tooling watching for Reconciling can observe the momentary
  absence. Rewrite the comment to name that trade-off explicitly and
  point at server-side apply as the fuller fix if the operator ever
  migrates to SSA globally.

* The unified own-marker/progressing/stuck gate in updateStatus is
  asymmetric on ObservedGeneration — isProgressingInRecovery checks it,
  isOwnMarker does not. That asymmetry is deliberate (a stale own-marker
  after a spec edit must still route to the state machine to avoid
  leaking the synthetic reason to the PackageSource — exactly the B2
  regression), not an oversight. Add a comment explaining the
  reasoning so a future edit doesn't "symmetrise" the check and re-open
  B2.

* forceArtifactGeneratorDrift's step-2 (metadata patch) failure was
  silent when it happened after a successful step-1 status patch. In
  that state source-watcher has been nudged, but the tracking
  annotations never persisted, so the next reconcile reads attempts=0
  and re-forces from scratch instead of accumulating toward
  SourceWatcherStalled — a persistent metadata-write-only apiserver
  malfunction would loop forever. Log the failure loudly with the
  intended attempt count so operators can spot this specific mode
  from cozystack-operator logs.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-05 08:30:19 +03:00
Ivan Okhotnikov 596427deb3 docs(operator): scrub stale awaitSourceWatcherResponse references from comments and test names
Self-review after the previous commit removed the awaitSourceWatcherResponse
helper (routing unified through maybeRecoverArtifactGenerator's state machine
via a single own-marker / progressing-in-recovery / stuck gate). Four
comments and two test names still referenced the now-deleted symbol —
misleading anyone tracing the architecture.

* maybeRecoverArtifactGenerator docstring: rewrote the trade-off paragraph
  to describe the unified gate and Wait branch instead of the removed
  helper.
* TestMaybeRecoverArtifactGenerator_DoesNotResetCounterOnFreshTransition
  comment: same rewrite.
* TestUpdateStatus_OwnMarkerRoutesToAwait renamed to
  TestUpdateStatus_OwnMarkerHoldsTrackingViaStateMachine + comment
  rewritten to describe the state-machine Wait branch.
* TestUpdateStatus_ProgressingDuringRecoveryRoutesToAwait renamed to
  TestUpdateStatus_ProgressingDuringRecoveryHoldsTracking + comment
  rewritten in parallel.

Pure doc / naming; no code or test-body changes. Tests still pass with
-race, vet clean, golangci-lint 0 issues.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-05 08:30:19 +03:00
Ivan Okhotnikov 7b08fcdca9 fix(operator): unify recovery gating through state machine, close source-watcher-unresponsive dormancy loop
Self-review on 467b788b8 caught a MAJOR bug introduced by the separate
awaitSourceWatcherResponse helper: if source-watcher never responds to
a forced drift (network partition, missing predicate, dead pod), our
own reasonRecoveryForced marker keeps reflecting back via Owns() forever,
and the wait-forever helper schedules a 1-second requeue tight loop
without ever advancing the retry state machine. SourceWatcherStalled
give-up was unreachable.

Fix: remove awaitSourceWatcherResponse and unify all three recovery
entry conditions — our own marker, source-watcher's Progressing during
rebuild, and the classic first-time stuck detection — through a single
gate that routes to maybeRecoverArtifactGenerator. The state machine's
decideRecovery is the single arbiter of wait/force/give-up, and its
Wait branch is bounded by the same backoff schedule as Force. When
backoff elapses on a still-reflecting own-marker the machine escalates
to another force (bumps attempts, re-stamps requestedAt); when attempts
exhaust it surfaces SourceWatcherStalled honestly.

The own-marker filter also no longer requires attempts>0, so an
operator crash between step 1 (status patch) and step 2 (metadata
patch) of forceArtifactGeneratorDrift is still routed through the
state machine on the next reconcile rather than leaking the synthetic
marker onto the PackageSource.

Also address the non-blocking notes from the same self-review:

* Docstring on maybeRecoverArtifactGenerator was still describing the
  removed reset-counter branch and the "requestedAt alone is a no-op"
  framing. Rewrite to explain the current shape: status write triggers
  the drift signal, requestedAt bump triggers the reconcile event —
  both are required, neither alone works.
* TestForceArtifactGeneratorDrift_MetadataPatchFailure_RollbackAnnotations
  now asserts reconcile.fluxcd.io/requestedAt is also rolled back on
  the metadata-patch failure branch (symmetric with the tracking
  annotations).
* TestUpdateStatus_OwnMarkerRoutesToAwait asserts an exact 25s
  RequeueAfter (backoffFor(1)=30s minus 5s elapsed) so a silent
  regression in the backoff arithmetic is caught.

New regression tests for the source-watcher-unresponsive path:

* TestUpdateStatus_OwnMarkerStale_TriggersRetry: with attempts=1 and
  lastRecoveryAt older than backoffFor(1), the reconcile must
  escalate through the Force branch and bump attempts to 2 rather
  than idle in a tight requeue loop.
* TestUpdateStatus_OwnMarkerAttemptsExhausted_GivesUp: with
  attempts=maxRecoveryAttempts and the marker still reflecting back,
  the reconcile must set PackageSource Ready=False /
  SourceWatcherStalled instead of extending the loop indefinitely.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-05 08:30:19 +03:00
Ivan Okhotnikov 934ca46a36 fix(operator): bump requestedAt with force-drift, filter own writes, remove reset-counter loop
Address lexfrei's three blockers on the force-drift approach — as
committed, the recovery driver did not function on a live cluster.

**B1**: `forceArtifactGeneratorDrift` wrote `Ready=False` on the AG's
status subresource but did not touch its metadata annotations.
source-watcher v2.1.0 registers its ArtifactGenerator watch with
`predicate.Or(GenerationChangedPredicate, ReconcileRequestedPredicate)`
— a status-only patch changes neither the generation nor
`reconcile.fluxcd.io/requestedAt`, so it does NOT enqueue a
source-watcher reconcile. The AG's periodic self-requeue is hardcoded
to `time.Hour`, well outside the 15m HR install timeout. Fix: also
stamp `reconcile.fluxcd.io/requestedAt` in the metadata patch so the
ReconcileRequestedPredicate fires and source-watcher's next reconcile
picks up the `IsFalse(Ready)` drift signal.

**B2**: `Owns(&ArtifactGenerator{})` re-fires this reconciler on the
force-drift's own writes. Both the status patch (Ready=False, reason
`SourceWatcherRecoveryForced`) and the metadata patch enqueue a
PackageSource reconcile. Blindly running `clearRecoveryTracking` on
those re-entrant fires nuked the attempts counter — the give-up
budget was unreachable, and the synthetic marker leaked to
`PackageSource.status.conditions[Ready]`. Fix: in `updateStatus`,
detect our own reason and route into a new `awaitSourceWatcherResponse`
helper that holds tracking and reports Unknown /
AwaitingSourceWatcherRecovery on the PS. The same helper covers the
Progressing/Unknown write source-watcher issues during rebuild once
tracking is present (attempts > 0) — otherwise the transient
Progressing would trigger the same premature tracking clear.

**B3**: The attempt-counter reset keyed on
`Ready.LastTransitionTime > lastRecoveryAt`. That inequality is true
on every source-watcher response during recovery (the Progressing
write is always newer than our force), so under the very race this
driver targets the counter would reset every cycle and
`recoveryActionGiveUp` was unreachable. Fix: remove the reset. The
canonical counter reset is the natural `clearRecoveryTracking` on the
not-stuck path (Ready=True from source-watcher, or a real upstream
Ready=False), and the HA/leader-election requirement already
contains the "missed recovery event on operator restart" edge case
the reset was trying to paper over.

Extract `reasonRecoveryForced` to a named constant so both the
writer (forceArtifactGeneratorDrift) and the reader (updateStatus'
own-marker filter) use the same symbol.

New tests pin all three fixes:
- ForceArtifactGeneratorDrift_Success now asserts the
  reconcile.fluxcd.io/requestedAt annotation is set (B1).
- UpdateStatus_OwnMarkerRoutesToAwait asserts we do NOT clear
  tracking and do NOT copy the synthetic Ready=False through when we
  see our own marker (B2).
- UpdateStatus_ProgressingDuringRecoveryRoutesToAwait covers the
  second Owns() re-entrancy path — source-watcher's Progressing
  write during rebuild (B2).
- UpdateStatus_ReadyTrueClearsTracking pins that the natural not-
  stuck path IS the canonical counter reset after B3.
- The old ResetsCounterOnFreshTransition test is replaced with
  DoesNotResetCounterOnFreshTransition, asserting the opposite
  behaviour that closes B3.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-05 08:30:19 +03:00
Ivan Okhotnikov f64081b232 fix(operator): schedule follow-up reconcile when Ready is Unknown within grace
E2E on dev3 caught a dormancy bug that renders the whole recovery driver
inactive in practice: when updateStatus copies Ready=Unknown across but
the AG's LastTransitionTime is still within stuckGracePeriod,
artifactGeneratorStuck returns false and we fall through to the copy
path with no RequeueAfter set. If the reconcile happens to fire in the
[stuck_start, stuck_start+30s) window — a routine case since operator
restarts touch every PackageSource — the controller then sleeps
waiting for an AG-status change that will never come (that IS the
fluxcd/pkg#934 lost-write race), up to the 10h informer resync.

Observed on dev3: `cozystack.networking` stuck at Ready=Unknown for 9+
minutes with zero recovery attempts. Two reconciles fired at
stuck_start+23s and +24s — both correctly detected "not yet stuck" —
and then nothing until observation ended. Upstream OCIRepository
revision was unchanged throughout, ruling out a coincidental source
update as the cause.

Fix: whenever we copy Ready=Unknown to the PackageSource (or set a
synthetic Unknown because no Ready condition exists yet), compute the
remaining grace period against the AG's LastTransitionTime (or
CreationTimestamp for the missing-Ready branch) and schedule a
RequeueAfter at that expiry. A 1s floor prevents a zero-delay tight
loop when the anchor is right at or past grace expiry.

Extracted the delay computation to `agFollowUpDelay` for direct unit
testing. Adds three tests: the follow-up scheduling on Ready=Unknown
in-grace, the same on the missing-Ready branch, and a table test on
agFollowUpDelay covering the floor edge cases.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-05 08:30:19 +03:00
Ivan Okhotnikov 4df18df5de test(operator): pin maybeRecoverArtifactGenerator branches, tighten wait assertions, polish comments
Address the fourth-pass self-review's non-blocking findings:

* Add TestMaybeRecoverArtifactGenerator_{ForceBranch, WaitBranch,
  GiveUpBranch, ResetsCounterOnFreshTransition}. The three switch
  outcomes of the driver (force / wait / give-up) and the reset-counter
  guard on a fresh Ready.LastTransitionTime transition were previously
  only covered indirectly through their pure-function components. Now
  each outcome is asserted end-to-end against a fake client with a
  status subresource, verifying the Ready condition, tracking
  annotations, RequeueAfter, and — for the wait branch — that no AG
  writes are issued at all.
* Tighten TestDecideRecovery's wait-branch assertions from a loose lower
  bound to exact equality. decideRecovery is deterministic
  (needed - elapsed) so an exact assertion catches subtle backoff
  regressions a lower bound would miss (coderabbit's inline nit).
* Rename the local var `wantWaitMin` to `wantWait` for the tighter
  assertion shape.
* Fix two stale identifier references in comments
  (maybeRequeueArtifactGenerator, AwaitingSourceWatcherRequeue) left
  over from the rename pass.
* Add a docstring paragraph on forceArtifactGeneratorDrift explaining
  why the JSON-merge-patch full-array-replace on
  `.status.conditions[]` is net-neutral in this workaround's context.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-05 08:30:19 +03:00
Ivan Okhotnikov 7c3af85e58 fix(operator): force source-watcher drift via status patch, not requeue bump
Rework the fluxcd/pkg#934 recovery driver after myasnikovdaniil's B1
finding: bumping reconcile.fluxcd.io/requestedAt cannot recover a stuck
ArtifactGenerator because source-watcher v2.1.0's reconcile is
drift-gated. detectDrift (artifactgenerator_drift.go:52) returns
NoDriftDetected for exactly the stuck signature we target — Inventory
populated, ObservedSourcesDigest set, Ready=Unknown on the current
generation — so the ReconcileRequestedPredicate fires a reconcile that
lands on the no-drift early-return at controller.go:164 and MarkTrue at
:254 is never reached. Every attempt in the previous shape was a no-op.

New mechanism: force drift by patching Ready=False on the AG's status
subresource. detectDrift picks this up as NotReady drift, source-watcher
takes the drifted branch, and either MarkTrue (on healthy rebuild) or
MarkFalse with a real reason (on genuine failure) replaces our synthetic
False. Owns() watch fires this reconciler to copy the resolved condition
through. Same bounded-retry harness (30s / 60s / 2m / 4m / 4m budget
inside the 15m HR install timeout, SourceWatcherStalled surfaced on
exhaustion), same annotation-based state on the AG, same rollback
contract on Patch failure — only the action itself changed.

Additional fixes bundled in:

* Reset attempt counter when the AG's Ready.LastTransitionTime is newer
  than our lastRecoveryAt, so a fresh stall episode after a previous
  give-up gets a fresh budget rather than an immediate re-give-up
  (myasnikovdaniil's follow-up #3).
* Propagate updateStatus errors from Reconcile instead of swallowing them,
  so controller-runtime's exponential backoff retries transient failures
  on force-drift / status-update writes rather than dropping them silently
  (coderabbit's inline finding).
* forceArtifactGeneratorDrift documents the two-subresource shape (status
  patch first, metadata patch second) and the partial-failure semantics.
* New tests cover the status-patch success / status-patch failure /
  metadata-patch failure paths, all with fake client + interceptor. The
  successful case asserts Ready=False actually lands in the apiserver via
  WithStatusSubresource.
* All annotation / action / decision / helper names renamed from
  *Requeue* to *Recovery* to match the new semantics.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-05 08:30:19 +03:00
Ivan Okhotnikov 4c3099d6f2 test(operator): pin overwrite-rollback contract, minor polish
Address the fourth-pass self-review's non-blocking NITs:

* Add TestBumpArtifactGeneratorRequeue_PatchFailure_RollbackRestoresPriorBump
  to cover the subtle case the two existing rollback tests miss: when the
  tracking annotations were already set from a prior bump, a failed Patch
  must restore the PRIOR values, not delete the keys. A refactor that
  naively swapped rollback for `delete(ag.Annotations, key)` would pass
  the nil-rollback tests but silently regress this contract.
* Rename the cloneAnnotations subtest from "deep copy" to "independent
  copy" — the code documents shallow copy semantics and the misnomer was
  the only thing pointing readers at the wrong mental model.
* Hoist `now.UTC().Format(time.RFC3339Nano)` to a local so the two
  timestamp annotations use one formatted value instead of two identical
  Format calls.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-05 08:30:19 +03:00
Ivan Okhotnikov 28834ca874 test(operator): cover rollback contract, deletion guard, and clone helper for the requeue driver
Address the third-pass self-review's blocker: the mutation contract on
bumpArtifactGeneratorRequeue / clearRequeueTracking, the DeletionTimestamp
early-return in Reconcile, and the nil-preserving cloneAnnotations helper
had no regression tests, so a future refactor could silently break the
contracts documented in comments.

New fake-client + interceptor-based tests:

* TestCloneAnnotations — pins the "nil in / nil out" sentinel behaviour
  and mutation-independence of the shallow copy.
* TestBumpArtifactGeneratorRequeue_Success — asserts the persisted state
  matches the caller's in-memory `ag.Annotations` on success, including
  that reconcile.fluxcd.io/requestedAt is actually written (previously
  never asserted anywhere).
* TestBumpArtifactGeneratorRequeue_PatchFailure_Rollback and its
  PreservesUnrelated variant — drive the Patch-failure branch via an
  interceptor client and assert both that our tracking annotations are
  rolled back and that pre-existing unrelated annotations survive.
* TestClearRequeueTracking_Noop — asserts we don't issue a no-op Patch on
  every healthy reconcile when there are no tracking annotations to clear.
* TestClearRequeueTracking_PatchFailure_Rollback — same rollback contract
  as bump, on the clear path.
* TestReconcile_DeletionTimestamp_EarlyReturn — asserts a PackageSource
  with a deletionTimestamp does not receive any writes and does not
  schedule a RequeueAfter.

Doc tightening in the same commit:

* Reword clearRequeueTracking's comment on the flux annotation semantics —
  standard flux controllers do not clear reconcile.fluxcd.io/requestedAt,
  they only advance .status.lastHandledReconcileAt via the
  ReconcileRequestedPredicate.
* Note controller-runtime v0.15+'s default RecoverPanic behaviour where
  the retry-driver's unhandled-action panic is raised, so a future
  SetupWithManager flip that disables it is easier to spot.
* Explain why cloneAnnotations' shallow copy is sound (annotation values
  are immutable strings).

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-05 08:30:19 +03:00
Ivan Okhotnikov c3f9125163 fix(operator): skip deleted PackageSource, restore ag mutation contract, document HA + timing rationale
Address second-pass self-review findings on the bounded-requeue driver:

* Skip reconcile early when the PackageSource has a deletionTimestamp so
  the retry driver stops writing "AwaitingSourceWatcherRequeue" onto an
  object about to be garbage collected. ownerReference cascade handles
  the ArtifactGenerator cleanup.
* Fix the mutation contract on bumpArtifactGeneratorRequeue and
  clearRequeueTracking: the previous deep-copy shape left the caller's
  `ag` desynchronised from the apiserver on success. Now mutations land
  on `ag` itself with a rollback to the pre-call annotation snapshot on
  Patch failure — so success reflects persisted state and failure does
  not leak unpersisted state to subsequent code paths.
* Convert the "unreachable" default in maybeRequeueArtifactGenerator to
  an explicit panic so a future requeueAction added without a matching
  case fails loudly instead of freezing the Ready condition silently.
* Document the timing constants' rationale (grace period matches the
  observed &lt;10s source-watcher settle time on dev3; total ~11.5m budget
  keeps SourceWatcherStalled inside the 15m HR install timeout) and note
  the operator must run with leader election when replicas > 1 so the
  annotation-based retry state does not race.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-05 08:30:19 +03:00
Ivan Okhotnikov c5845db221 refactor(operator): thread single now through reconcile, harden patches, tighten status
Address self-review nits on the bounded-requeue driver:

* Sample `now := time.Now()` once at Reconcile entry and thread it into
  `updateStatus`, `artifactGeneratorStuck` and `maybeRequeueArtifactGenerator`
  so every time-sensitive check inside a reconcile agrees on the same
  wall-clock reading.
* Set `ObservedGeneration` in the three early Ready-condition branches
  (`SourceRefNotSet`, `ArtifactGeneratorNotFound`, `ArtifactGeneratorNotReady`)
  for parity with the retry-driver branches.
* Patch the ArtifactGenerator via a deep copy in
  `bumpArtifactGeneratorRequeue` and `clearRequeueTracking` so a failed Patch
  no longer leaves the caller's in-memory `ag` desynchronised from the
  apiserver.
* Document the retry driver's dependency on source-watcher's
  ReconcileRequestedPredicate on ArtifactGenerator, so a future upstream
  change dropping the predicate is easier to trace.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-05 08:30:19 +03:00
Ivan Okhotnikov ff314797e9 refactor(operator): drive source-watcher via bounded requeue instead of synthesising PackageSource Ready
Replaces the Ready=True synthesis for PackageSources whose ArtifactGenerator
hit the fluxcd/pkg#934 patch.Helper split-write race with a bounded retry
driver: on detected stall, bump reconcile.fluxcd.io/requestedAt with
exponential backoff (30s to 4m cap) up to 5 attempts, then surface as
PackageSource Ready=False with reason SourceWatcherStalled.

The previous approach could mask a genuine regeneration failure if the same
patch.Helper race that the workaround exists for also swallowed a real
Ready=False write, and it permanently reinterpreted PackageSource.Ready
semantics from "current revision processed" to "valid consumable artifacts
exist". Requeue keeps the CRD contract intact and, on stall exhaustion,
produces an honest failure signal an operator can act on instead of a
false-positive True.

Retry state (attempts + last-bump timestamp) lives on the AG as
cozystack.io/source-watcher-* annotations so it survives operator restarts.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-05 08:30:19 +03:00
Ivan Okhotnikov 88d62612fe docs(operator): correct meta.SetStatusCondition rewrite semantics in comment
review-branch skill flagged that the comment claimed
meta.SetStatusCondition rewrites LastTransitionTime on Status OR
Reason change. Actually only Status change triggers a
LastTransitionTime rewrite — Reason and Message are updated in
place, LastTransitionTime is preserved. Outcome (repeated
identical calls are no-op-y on persisted state) is unchanged;
just tightening the technical claim to match apimachinery.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-05 08:30:19 +03:00
Ivan Okhotnikov 122584045b docs(operator): link workaround to fluxcd/pkg#934 (existing upstream issue)
Reviewer nit was to file an upstream issue so the workaround does
not silently outlive its cause. Checked first — fluxcd/pkg already
has one open: #934 \"patch.Helper creates race conditions in
downstream controllers\" (May 2025). Same root cause we traced:
patch.Helper writes .status and .status.conditions as separate
apiserver requests, so an etcd blip between the two leaves the
object half-persisted. The upstream issue is broader than just
source-watcher — every fluxcd downstream controller consuming
fluxcd/pkg's runtime patch helper is affected.

Update the TODO(remove) marker on our workaround to point at
that issue and note that both source-watcher v2.1.0 and v2.2.x
(flux-aio latest) pull the same fluxcd/pkg runtime, so bumping
source-watcher alone does not remove the need for this synthesis.
Compile + unit tests unchanged.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-05 08:30:19 +03:00
Ivan Okhotnikov 2fcf0bf5be fix(operator): document PackageSource Ready contract change + address review nits
kvaps and lexfrei confirmed the diagnosis by tracing source-watcher
v2.1.0 (same code paths in v2.2.x / flux-aio latest, so a simple
version bump does not remove the need for this workaround):

  * fluxcd's patch.Helper patches .status and .status.conditions as
    SEPARATE apiserver requests, so an etcd timeout during first-
    install load can land between them — leaving Inventory /
    ObservedSourcesDigest persisted while the Ready-condition write
    is lost.
  * On the next reconcile the !hasDrifted branch returns before
    touching conditions. The deferred summarizeStatus still runs and
    adjusts the Reconciling condition, but it never promotes a
    stuck Unknown Ready to True — that promotion only happens on the
    successful hasDrifted path.

Two substantive changes prompted by the reviews:

  1. Document the semantic change the workaround introduces on
     PackageSource.status.conditions[Ready]. fluxcd/pkg/runtime/
     conditions.Set writes condition.ObservedGeneration =
     ag.Generation on EVERY condition mutation including the
     Progressing/Unknown mark at the start of a rebuild. And
     ArtifactGeneratorStatus does not expose an object-level
     ObservedGeneration (only ReconcileRequestStatus is inlined).
     So Generation-matching cannot distinguish 'current revision
     fully processed' from 'rebuild in progress, previous
     Inventory/digest still persisted' — meaning the predicate can
     (and does) fire during a content-only regeneration (new OCI
     revision, same .metadata.generation).

     Net effect: PackageSource Ready no longer dips through Unknown
     while a new revision is being processed; it now means 'valid
     consumable artifacts exist' rather than 'the current revision
     has been processed'. Kvaps flagged this as the better contract
     for install/upgrade ordering (no InProgress flapping, and a
     genuine regeneration failure still surfaces as Ready=False on
     the successful hasDrifted path). Document it inline instead of
     implying the workaround only fires in the stuck-status window.

  2. Reworked the predicate signature to take the pre-resolved
     Ready condition as a parameter. The caller was already doing
     the FindStatusCondition lookup for the pass-through path, so
     the helper no longer duplicates it (kvaps' nit).

Also refreshed the pre-existing comments:
  * The 'early-exits before touching status again' step is now
    correctly worded as 'returns before touching conditions again;
    the deferred summarizeStatus never promotes a stuck Unknown
    Ready to True'.
  * Bug attribution now names the code paths ('!hasDrifted branch',
    'summarizeStatus deferred'), notes that v2.1.0 and v2.2.x
    ('flux-aio latest') both ship the affected code, and carries a
    TODO(remove once upstream is fixed) marker so the workaround
    cannot silently outlive its cause.

Tests: existing 8 subtests updated to construct the ready condition
manually so the call matches the new signature, plus one new
subtest locking in the intentional 'content-only regeneration
predicate fires' behaviour so a future refactor that tightens the
predicate back to 'current revision processed' — which would
re-introduce the flapping — fails loudly. 9/9 pass.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-05 08:30:19 +03:00
Ivan Okhotnikov a03e879316 fix(operator): require Ready.ObservedGeneration == ag.Generation before synthesising
Own-audit found the first pass had a spec-update race: if a user
edited PackageSource.spec.variants on a running cluster, cozystack
would bump the derived ArtifactGenerator to Generation=2, and
between "source-watcher started reconciling gen=2" and "source-
watcher wrote new Inventory + ObservedSourcesDigest for gen=2" the
ArtifactGenerator still exposes the previous generation's
Inventory / ObservedSourcesDigest AND Ready=Unknown (Progressing).
The workaround would synthesise Ready=True on stale artifacts, and
downstream Flux would consume the previous-generation
ExternalArtifacts as if the new spec had converged.

Tighten the predicate to require the Ready condition's
ObservedGeneration to match ag.Generation. Only when source-watcher
has actually processed the current spec (which happens strictly
after it writes the current-generation Inventory + Digest) does the
Ready condition carry ObservedGeneration == ag.Generation. That is
exactly the window where Inventory / Digest are guaranteed current-
spec AND only the status patch on Ready itself failed — the actual
upstream bug this workaround exists to paper over.

Consequence: if Ready is missing entirely we return false now
(was true). We cannot verify ObservedGeneration on an absent
condition, and the missing-Ready state is indistinguishable from
mid-reconcile-of-a-newer-generation. On the next reconcile source-
watcher will write at least a Progressing condition, at which point
the current-spec Generation check will be evaluable and the
workaround will kick in if — and only if — Inventory / Digest also
match the current spec.

Tests updated:
  * changed 'no Ready condition' case from want=true to want=false
    (reflecting the tighter invariant);
  * added a case where Ready=Unknown carries an OLD Generation
    (want=false — do not synthesise on stale artifacts);
  * stamped ObservedGeneration on every existing condition-bearing
    case so they exercise the matching-Generation branch;
  * introduced a small agWithGen helper to keep the fixtures
    readable now that Generation is load-bearing.

All 8 subtests pass; existing operator suite unaffected.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-05 08:30:19 +03:00