mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
df889a64193cacdf922f3de292fa3e37cc8047b6
79
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
30ee57f892 |
fix(operator): route every downward ceiling move through the grace period
The grace period guarded only the branch where the scan came back empty, which left the same defect standing in the branch where it finds something smaller than the ceiling already in place. A namespace raised to 16Gi for a pod visible only as a live Pod, with a second oversized pod at 8Gi beside it, loses the larger pod to a node drain. The scan still finds the 8Gi one, so the empty-scan guard never engages, and the ceiling was written straight down to 8Gi -- rejecting the larger pod's recreation exactly as writing the configured limit would have. A blocker smaller than the standing ceiling is not evidence that a larger one is gone. So the rule is now about direction rather than about why: the ceiling the evidence justifies is computed first, and any move that lowers it goes through the grace period, whichever of the three reasons prompted it -- an empty scan, a smaller blocker, or the configured limit itself being lowered beneath a standing ceiling. The eventual drop lands on what the scan justifies rather than on the configured limit, so a smaller blocker that is genuinely still there keeps the ceiling it needs. The raise is also no longer logged when it changes nothing. A namespace holding a permanently oversized pod re-justifies the same ceiling on every reconcile, and announcing that put a line in the log for every such namespace on every HelmRelease status change. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
9fe93a3d7e |
docs(operator): fix three claims that overstated what the code does
The static-pod note said containers built from an on-disk manifest are unaffected "in both directions". True of the container, which the kubelet runs from disk whatever the API says, but the mirror pod is an ordinary API object that passes admission like any other, so a static pod declaring a memory request above the ceiling with no limit of its own has its mirror pod rejected -- leaving the container running with no representation in the API. Worth naming, since kube-system is in scope by design. The scan does cover it: a mirror pod is a live Pod. The ReplicaSet exclusion was justified on live ones being copies of their Deployment's template. That does not reach an ownerless ReplicaSet, which at replicas 0 has neither pods to be seen through nor a Deployment template standing in for it. It stays out of scope -- nothing here ships a bare ReplicaSet -- but as a gap left open rather than one the stated reason closes. The delete guard's comment claimed the enabled apply always adopts and stamps a same-named foreign object. It does not when that object's spec already matches, because then the apply is skipped entirely. The outcome is the same one the guard gives on purpose, but it is reached by the apply not running. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
e131a49514 |
feat(operator): default the system namespace memory ceiling to 32Gi
The guard in front of this default reads declared memory requests. It cannot see a container's real working set, so a system component that requests little but legitimately uses more than the ceiling, and declares no limit of its own, was handed a deterministic kernel OOM on its next recreation in exchange for leaving the Talos handler's victim set. Since the feature ships enabled, that trade fell on the whole fleet on upgrade. It is not a trade this feature has to make. The handler discards a cgroup for carrying a memory.max at all, whatever its value, so immunity does not require a fitted ceiling -- only a real one. A default far above any plausible working set buys the immunity and leaves nothing for the ceiling to bind on, which also keeps it clear of the failure mode this repository already carries a scar from, where an undersized limit converted a rare pressure-driven kill into a reliable one. Nothing about scheduling changes: only requests are scheduled against, and the paired default request stays 32Mi. An operator who wants the ceiling to bind lowers it, per component in a chart or fleet-wide with this knob, once real usage is measured -- which is the direction that needs evidence, and now the direction that needs it rather than the default. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
ad3c9ef04e |
fix(operator): stay out of a namespace another LimitRange governs
The reconcile only read its own LimitRange by name, so a system namespace already carrying an administrator's LimitRange got a second one beside it. LimitRanger applies defaults from every LimitRange in the namespace and then validates the pod against every one, and nothing orders them, so the effective container ceiling moved between the administrator's value and this one for reasons no chart author could see. That is the same hazard this feature cites to keep itself out of tenant namespaces, and it does not stop being one because the namespace is a system namespace. A max is the sharper half. This reconciler writes a default and no max, so an administrator's LimitRange carrying a container memory max below the configured default had every container that declares no limit of its own rejected: the default is written first, then validated against the max it exceeds. That takes out a whole namespace, and it reaches kube-system, which this feature enters for cozystack-scheduler and which is where a distribution's or an administrator's own LimitRange is likeliest to sit. Such a namespace is now left alone and logged, and anything this reconciler wrote there earlier is withdrawn rather than left sitting beside the foreign object, since that object is half of the ambiguity. The check is deliberately narrow -- only a container memory default or max collides with what is written here -- because read more widely it would be a blanket opt-out that an unrelated cpu policy could trip. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
5630954e34 |
fix(operator): hold a raised memory ceiling before lowering it
A raised container memory ceiling was recomputed from the scan on every reconcile, so a single scan that found nothing lowered it back to the configured limit. An empty scan is not evidence that the oversized request is gone: it is equally consistent with the pod being between deletion and recreation. For a blocker that appears only as a live Pod, which is a class this repo ships -- every CNPG Cluster in a system namespace builds its instance pods directly rather than through a StatefulSet -- an ordinary Talos node drain opens that window, and reconciles follow HelmRelease status churn, of which a node upgrade produces plenty. The ceiling was lowered under a pod that was merely being rescheduled; its recreation was then defaulted a limit below its own request and rejected at admission for good, because the rejected pod never exists so no later scan can find it, and the namespace settles at the configured limit where the pod half of the scan stops running at all. The drop was not even logged. Lowering now waits until the same emptiness has held for memoryCeilingDropGrace, with the instant it was first observed recorded on the object so it survives an operator restart and a change of leader. A wall clock rather than a count of consecutive clean scans, because the reconcile rate rises exactly when the risk does and any small count can be burned inside one rescheduling window. A blocker seen again cancels the countdown, an unreadable stamp restarts it, and both the hold and the drop are logged. The steady-state comparison now also covers the two metadata fields this reconciler manages, because the grace stamp is one of them. That incidentally fixes the managed-by label never being re-stamped when only metadata had drifted, which had left turning the knob off unable to remove an object the operator did own. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
3c1d8d6988 |
docs(operator): bound the managed-by guard to the disabled state
The comment on the label check in deleteSystemDefaultsLimitRange reads as a general promise not to touch an administrator's own LimitRange of that name. It is narrower than that, and the difference is worth writing down: while the feature is enabled the apply takes the name over with ForceOwnership and stamps the managed-by label itself, so a hand-written cozystack-system-defaults is adopted and then deleted here when the knob goes off. That is the intended SSA behaviour, not a defect. What the label actually buys is that a LimitRange of that name the reconciler never wrote to survives the knob, which is a real guarantee and a smaller one. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
accb4f9e1b |
refactor(operator): drop the dead error check from the settled-state guard
The guard read `err == nil && apiequality.Semantic.DeepEqual(...)`, which looks like it consults the earlier Get and does not. That Get's error was reassigned by the scan two statements up, and the scan's non-nil case has already returned, so the clause is always true by the time it is evaluated. It was harmless only by accident: a namespace whose Get returned NotFound carries an empty spec, which never compares equal to the desired one, so the apply happened anyway. Removing the clause makes that the reason rather than a coincidence sitting behind a condition that cannot fire. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
ead330ccc3 |
fix(operator): keep scanning pods while the ceiling stays raised
The pod half of the memory scan was skipped whenever any container memory default was in force. That is the weaker of the two available conditions, and it silently retracted a ceiling this reconciler had just raised. The default written into a system namespace is either the configured limit or a ceiling raised above it to clear an oversized request, and the raise is recomputed from the scan every reconcile rather than read back off the object. So a raise that only the pod half could justify -- a live pod requesting more than the limit with no limit of its own, in no workload template the scan can read, which is what an operator building pods from its own custom resource produces -- lasted exactly one reconcile. The raised LimitRange satisfied "in force", the next reconcile skipped the pod list, the scan came back empty, and the configured limit was reapplied underneath a still-running pod. That pod's next recreation was rejected with "must be less than or equal to memory limit", and because the skip stayed in effect nothing ever raised the ceiling again: a healthy component goes down at a node reboot and stays down, with the revert unlogged. Lowering the knob reached the same place with no race in it. An operator running at 16Gi has pods admitted with requests only that ceiling allowed; dropping the flag to 4Gi wrote 4Gi straight over them. Gate the skip on the namespace already defaulting memory at exactly the configured limit instead. That keeps the pod list alive for as long as a namespace sits raised, which is what lets the raise be re-justified or dropped on evidence rather than frozen or forgotten, and it still skips the list in the steady state the skip was added for. The comparison is by value for the same reason the spec comparison is semantic: 4096Mi and 4Gi are one ceiling. The cost is that a raised namespace reads the pod list every reconcile. That is the price of being able to lower the ceiling again, bounded by how many namespaces sit raised -- none, once the oversized requests are gone. Three tests cover it, each of which fails on the old condition: a raised ceiling held while the pod needing it runs, the same ceiling dropping back once that pod is gone, and the configured limit falling below a ceiling already in place. A fourth pins the notation equality. The fixture comment on the steady-state test claimed its 8Gi pod could not exist beside a settled 4Gi LimitRange; it can, through a narrow window, so it now says so. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
26a1ce5be0 |
test(operator): drop the incidental fixture from the no-pods case
The scaled-to-zero test seeded a spec-less LimitRange, which is not part of the case it names. It is the first application in the namespace that makes a workload with no pods invisible to a pod-only scan, so start from no LimitRange at all and let the assertion rest on the workload rather than on the fixture. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
b62fac474c |
refactor(operator): move both server-side applies off the deprecated client.Apply
client.Apply as a patch type is deprecated in controller-runtime v0.23.1 in favour of Client.Apply taking a runtime.ApplyConfiguration. Convert both call sites — the system defaults LimitRange, new on this branch, and the Namespace apply that predates it — keeping the same field owner and force-ownership options. The LimitRange keeps its typed builder, because that is what the spec comparison and the tests read, and gains a small function that restates it as an apply configuration immediately before the wire. The Namespace apply now sends only the labels and annotations the reconciler actually sets, rather than a whole typed object whose spec and status were empty. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
c7af79b734 |
fix(operator): only delete the LimitRange this reconciler owns
deleteSystemDefaultsLimitRange matched on name alone, and the name is not reserved: nothing stops an administrator creating a LimitRange called cozystack-system-defaults by hand. Turning the knob off would then silently remove policy this operator never wrote, in every system namespace at once. Stamp app.kubernetes.io/managed-by on the object the reconciler applies and require it before deleting. Anything else is left where it is, which is also the right answer if the name is ever taken over by another component. Nothing in the wild carries the old unlabelled shape, since the feature has not shipped. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
dbc5e5ea13 |
perf(operator): read before deleting the disabled LimitRange
With the limit set to 0 the disabled path runs for every system namespace on every Package reconcile and issued a DELETE each time, swallowing the 404 that nearly all of them returned. That is a write attempt per namespace per reconcile, forever, and every one of them lands in the audit log. Get from the informer cache first and return when there is nothing there. The test now counts deletes, so a second reconcile over an already-empty namespace has to stay at zero rather than merely not erroring. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
89b78cbf20 |
docs(operator): drop the duplicated field from the ceiling-raise log
request and raisedLimit were logged with the same value, because the raised ceiling is the blocking request. Keep request, which names the thing an operator has to go and change, and say in the message that the ceiling becomes it. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
87f79c17c3 |
fix(operator): keep scanning templates after the LimitRange settles
The steady-state fast path returned as soon as the LimitRange matched the configured spec, on the grounds that LimitRanger gates anything a scan could still find. That holds for pods and not for templates: a Deployment added to a settled namespace with a memory request above the ceiling and no limit of its own is admitted exactly as written, and only the pods it then tries to create are rejected with "must be less than or equal to memory limit". Nothing raised the ceiling, because every later reconcile returned at the same line, so a chart bump became a rollout that could never produce a pod and logged nothing about it. Split the scan instead. Templates are read on every reconcile; the pod list, which is the expensive half, is read only while no memory default is in force in the namespace — once one is, LimitRanger fills in a limit for every container admitted without one, so a container carrying a request with no limit beside it can no longer come into existence. The skip keys on the default actually being in force rather than on the object existing, so a LimitRange whose spec was stripped scans pods again. The write is now guarded by a semantic comparison against the spec the scan settled on rather than against the configured spec, so a namespace already sitting at a raised ceiling stops taking an identical apply on every reconcile as well. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
c9cacda1a0 |
docs(operator): pin the admission order by rule, not plugin index
The scan's justification for treating a VerticalPodAutoscaler as harmless rested on LimitRanger being "position 6 of AllOrderedPlugins" and MutatingAdmissionWebhook being "35". Both numbers are properties of one release rather than of the ordering, and they had already drifted out from under the comment. Under k8s 1.35, which this tree builds against, LimitRanger is the 6th entry but MutatingAdmissionWebhook is the 33rd; index 35 is ResourceQuota and the 35th entry is ValidatingAdmissionWebhook, neither of which mutates, so the sentence named the wrong plugin under either counting convention. In 1.31 the same plugin sat at 30, and SecurityContextDeny being dropped in 1.35 is what moved LimitRanger. Restate it on the invariant the list asserts about itself -- webhook, resourcequota and deny plugins must go at the end -- with LimitRanger compiled in above that marker and every plugin that can rewrite a container's resources below it. That survives a release bump where an index does not, and it covers MutatingAdmissionPolicy as well, a second mutator that the positional wording predated and never accounted for. Record what the order buys, which the old wording left implied. Run the other way round, a plugin mutating before LimitRanger would write a request into a container that has no limit yet, LimitRanger would then default the limit to the configured ceiling, and core validation would reject the fully mutated pod. The scan can see none of that -- the request is in no template and the pod is never admitted -- so the order is the only thing keeping VPA out of the residual gap this comment already records for controlledValues: RequestsOnly. Fix the matching rationale on the live-pod case in TestFindRequestAboveDefaultLimit, which still claimed the pod's request was raised by a webhook after the template was rendered. The ordering rules that out; what the case actually rests on is a pod admitted before the LimitRange existed, or one its controller builds from a custom resource no template in the scan can show. No behaviour change, and no new test: a container that carries its own memory limit is already pinned out of the scan's scope for a template, in TestFindRequestAboveDefaultLimit, and for a live pod, in TestReconcileSystemDefaultsLimitRangeAppliesWhenLargeRequestCarriesItsOwnLimit. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
16b8ea4f0e |
docs(operator): correct the VPA rationale on the request scan
The comment argued that the danger this scan guards against is a VPA webhook writing a request above the defaulted limit, which is then rejected at admission. Admission ordering rules that out. LimitRanger is position 6 of AllOrderedPlugins and MutatingAdmissionWebhook is 35, so the LimitRange has already defaulted the container's memory limit before VPA is called, and VPA rescales that limit in proportion to the request it writes (GetProportionalLimit, under the default controlledValues: RequestsAndLimits). Request and limit stay consistent and the pod is admitted. Rewrite it around what can actually happen: a request already in the spec when it reaches admission with no limit beside it. In this tree that means an operator lowering --system-namespace-memory-limit below a statically declared request -- the largest is 2Gi, in packages/system/rabbitmq-operator, so the 4Gi default clears them all today. Two narrower shapes are real too: a pod that predates the LimitRange, whose own controller may build it from a custom resource no workload template in this scan can show; and a VPA set to controlledValues: RequestsOnly, where nothing rescales the defaulted limit and the rejection returns. The second doubles as the residual gap, since a first pod rejected outright appears in neither a template nor a pod list. The scan itself is unchanged; only the justification was wrong. Also record why withholding the LimitRange had to go rather than merely being suboptimal: in a namespace with no LimitRange nothing defaults a memory limit, so the next VPA-sized pod arrives with a large request and no limit, the scan finds it again, and the namespace is withheld again. The old behaviour could never get its foot in the door in cozy-monitoring. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
d682158d1f |
test(operator): pin the tenant namespace boundary
One strings.HasPrefix keeps this feature out of tenant namespaces, and nothing tested it: "tenant-" appeared twice in package_reconciler.go and not once in the test file, so replacing the check with isSystem := true left the whole package green. What it protects is not subtle. packages/apps/tenant ships tenant-range-limits, which defaults container memory to 128Mi in every tenant namespace that declares resourceQuotas. A system LimitRange landing beside it leaves two LimitRanges defaulting the same containers, and which of them wins is decided by the order LimitRanger happens to iterate them -- neither chart controls that -- so a tenant container's default memory limit would move between 128Mi and 4Gi with nothing in the tree changing. Reconcile a variant with one system and one tenant namespace, and assert no LimitRange is written for the tenant one. The system namespace in the same run is the positive control: without it the test would still pass if LimitRanges stopped being written anywhere at all. The cozystack.io/system label is asserted too, since the same boundary gates both. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
2e1ab6682c |
fix(operator): raise the namespace ceiling instead of retracting it
When a container in a system namespace requests more memory than the configured default, the guard used to withhold the LimitRange and delete any it had applied earlier. That leaves the namespace with no default memory limit at all, which is the precise state this feature exists to remove: without memory.max on the pod cgroup the Talos OOM handler keeps the pod in its victim set, and a tenant workload's memory pressure gets paid for by a system component. Default that namespace's memory to the offending request instead. A looser ceiling still sets memory.max, so the pod scores zero in victim selection exactly as it would at the configured limit -- 8Gi protects it as well as 4Gi does, and both protect it infinitely better than nothing. The raise needs no knowledge of where the request came from, and it undoes itself: once the request is gone the scan finds nothing, desired falls back to the configured limit, and the next reconcile applies it. This is not a hypothetical namespace. packages/system/monitoring pins maxAllowed.memory at 8Gi for vmselect and vmstorage, packages/system/monitoring-agents pins 6G for vmagent, all three run updateMode: Initial, and no monitoring template declares a memory limit. Initial never evicts, so those requests do not come down on their own: under the old behaviour cozy-monitoring, which holds the largest memory consumers on the platform, was permanently excluded from the feature on exactly the clusters it was written for. A raised namespace no longer matches the base desired spec, so it scans on every reconcile. That is deliberate: the raise has to track the current largest request in order to be able to drop back, and caching it would freeze a ceiling that nothing would ever lower again. The set of raised namespaces is small. The log moves from Error to Info. Nothing is wrong when this fires; the operator is reporting a ceiling it chose. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
c65db1ad52 |
perf(operator): skip the memory scan when the LimitRange is settled
reconcileSystemDefaultsLimitRange runs for every system namespace on every Package reconcile, and Package reconciles follow HelmRelease status churn through Owns, so on a platform with dozens of system namespaces the pod and workload scan behind the guard is by far the most expensive thing in the function -- and it is uncached by design, because caching it would mean a cluster-wide Pod informer in an operator whose job here is to bound memory. Build the desired LimitRange first, read the existing one from the informer cache, and return without scanning when the two specs already match. That is safe because the LimitRange being applied is itself the guarantee: from that moment LimitRanger rejects any new pod whose memory request exceeds this default, so the only requests a scan could still find are ones that predate the LimitRange, on pods already running, and there is nothing it would do differently about them. Steady state becomes one cached GET. The comparison is apiequality.Semantic, not reflect.DeepEqual, because resource.Quantity carries its parsed format and cached string: 4Gi and 4294967296 are the same ceiling and must not read as drift, or every namespace would scan and re-apply forever. The test counts List calls rather than only checking the resulting LimitRange, which cannot tell a skipped scan from a repeated one that reached the same answer, and seeds an 8Gi pod so a scan that ran would also change the object. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
b55c4f347a |
docs(operator): keep the residual-gap note on the request scan
The pod-only scan documented what it could not see: a request that only ever exists post-admission, on a workload whose first pod is rejected outright. Reading pod templates narrows that gap rather than closing it, and the rewritten comment dropped the note entirely, which reads as a claim that nothing is left. Restore it in its narrower form: a template under the limit, a webhook that raises the request past it, and a first pod rejected before it exists, is still invisible to this scan. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
10acfe025d |
test(operator): cover both terminal pod phases in the skip
findRequestAboveDefaultLimit skips a pod in PodSucceeded or PodFailed, because neither is ever recreated from that spec. Only the Succeeded half had a fixture, so deleting the PodFailed clause left the whole ./internal suite green -- half the condition was uncovered. Give the case a Failed pod as well. Deleting either clause now fails it. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
654412acd2 |
fix(operator): scan workload templates for oversized memory requests
reconcileSystemDefaultsLimitRange withholds the LimitRange when a container in the namespace requests more memory than the configured default limit, because LimitRanger would then default a limit below that request and the API server rejects the pod. The scan behind that decision listed pods only, so it could not see a workload that currently has no pods: a Deployment or StatefulSet at replicas: 0, a CronJob between runs, a Job with spec.suspend. Those are ordinary steady states rather than corner cases. The namespace looked safe, the LimitRange was applied, and the pod was rejected at the next scale-up or the next schedule with "must be less than or equal to memory limit of 4Gi" -- a workload that cannot come back, discovered at the moment somebody needs it. Scan the pod templates of Deployments, StatefulSets, DaemonSets, CronJobs and Jobs as well as live pods. Both halves are needed: a template is the only place a scaled-to-zero workload appears at all, and a live pod is the only place a request that the VPA admission webhook raised after admission appears. ReplicaSets are deliberately left out -- a live one copies its Deployment's template, and the older revisions a Deployment keeps carry superseded templates that only an explicit rollout undo would ever instantiate, so scanning them would withhold the LimitRange over a request replaced releases ago. Jobs are named separately because not every Job has an owner whose template would cover it, and a finished one is skipped for the same reason a terminated pod is. A suspended CronJob is not skipped: suspend is the same dormancy as replicas: 0, undone by the same kind of human action, and exempting it would leave the gap open for one kind. Templates are scanned before pods and the comparison stays strict, so a workload and its own pods report once, as the workload -- the object an operator has to edit, and one that outlives pod names. A live pod outranks its own template only when its request is strictly larger, which is exactly the webhook-injected case. memoryRequestBlocker therefore carries Kind/name instead of a bare pod name, and the log key is "workload" rather than "pod". Every added List fails closed, which the caller already relies on: a scan that cannot read is not evidence that applying is safe, so the namespace is skipped and logged instead. The test for that seeds a sentinel LimitRange, because asserting the object still exists cannot tell "left alone" from "applied blind" -- a swallowed List error applies a fresh LimitRange that exists just the same. RBAC needs no chart change: the operator's ServiceAccount is bound to cluster-admin, so it can already list these kinds. The kubebuilder markers are extended anyway to record what the controller reads. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
7cc090b05f |
test(operator): pin the largest-offender scan to all three fields
The "largest offender is reported" case asserted only the container name, and systemPod named every container "app", so the assertion held whichever of the two pods findRequestAboveDefaultLimit had picked. The comparison that keeps the largest request was uncovered. Name the fixture container after its pod, assert the reported pod, container and quantity together, and give the case three offenders whose largest sits in the middle of the name-ordered pod listing, so keeping the first, the last or the smallest offender all name a different pod and quantity. The init-container case now pins the same three fields. No production behaviour changes. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
546d8e6a54 |
feat(operator): default system namespace memory via LimitRange
The Talos userspace OOM handler (v1.12+) picks its victim by ranking
cgroups with `memory_max.hasValue() ? 0.0 : {Besteffort: 1.0, Burstable:
0.5, ...}[class] * memory_current`, discarding every cgroup that scores
zero. Only a memory limit sets memory.max on the pod cgroup, so only a
limit takes a pod out of the candidate set: a request merely demotes
BestEffort to Burstable, which under the v1.13.6 default
strictCgroupClassOrdering still means "killed second". PriorityClass has
no effect at all, because the handler walks cgroupfs and never talks to
the API server.
Victim selection is decoupled from the trigger, so the cgroup that
caused the pressure and the cgroup that gets SIGKILLed are unrelated by
design. Tenant workloads generally carry limits and system components
generally did not, which inverted the intended order: a tenant workload
thrashing against its own limit drives node-wide memory PSI, and
metallb-speaker or linstor-satellite is killed for it while the workload
responsible is never touched.
An audit found roughly 70 workloads across ~30 packages with at least
one container missing a memory limit, so per-chart resources would
neither cover the current set nor keep covering it. Maintain a
LimitRange in every system namespace instead, at the single choke point
that already creates them and already distinguishes system from tenant
(reconcileNamespaces). Tenant namespaces are left alone because
packages/apps/tenant owns LimitRange policy there.
The limit is a ceiling, not a reservation: it defaults well above real
usage (4Gi against a largest observed system request of 512Mi).
defaultRequest is set explicitly and small, because Kubernetes otherwise
defaults each request to the limit and reserves that much at schedule
time. Setting the limit to 0 disables the LimitRange and removes any the
operator previously created.
Two things keep the feature from being able to hurt more than it helps.
A LimitRange default only reaches a container that declares no memory
limit of its own, and the API server then rejects a request above its
limit. So before applying, scan the namespace for a container that
requests more than the configured limit and sets none, and where one
exists skip the namespace, retract any LimitRange applied earlier, and
log the pod, container, request and limit. This is not hypothetical:
packages/system/monitoring and packages/system/monitoring-agents ship
VPAs for cozy-monitoring with maxAllowed.memory of 8Gi and 6G under
updateMode: Initial, so the VPA admission webhook -- which runs after
the in-tree LimitRanger -- can write a request past 4Gi that no chart in
the tree declares. That is why the check runs against live pods rather
than rendered templates, and why it reads through APIReader rather than
starting a cluster-wide Pod informer.
And no failure inside the feature propagates: a LimitRange error, or a
pod list that cannot be read, is logged and stepped over instead of
failing the Package reconcile. A namespace without a LimitRange is back
to the behaviour of every earlier release, while a namespace that never
reconciles leaves its components uninstalled.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
|
b0e07cadbb |
fix(operator): rebuild each changed ArtifactGenerator generation once (#4594)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does After an upgrade, an ArtifactGenerator can report Ready on its new spec while its artifacts were built from the old one, and source-watcher never rebuilds them. That is how the bucket-system chart in #4593 shipped without cozy-lib. The cause is upstream (fluxcd/source-watcher#384). source-watcher builds from the spec it read when the reconcile started, but the final status patch stamps Ready with the live generation. An upgrade changes the packages source and the copy rules within seconds, so a spec change often lands during a build. The generator status has nothing that tells a stale build from a good one. So the operator now rebuilds every changed generation once, after source-watcher first reports it Ready, through the force-drift path the operator already uses for fluxcd/pkg#934. An unchanged spec rebuilds to the same digest, so ExternalArtifacts and HelmReleases do not move. Generation 1 is skipped, since the race needs a spec change during a build. On the first rollout every generator past generation 1 gets one extra rebuild, and its PackageSource shows `Unknown/AwaitingSourceWatcherRecovery` until that rebuild lands. Nothing waits on that condition today. The same rebuild repairs clusters already stuck like #4593. The recovery messages on the PackageSource no longer blame fluxcd/pkg#934, because the same path now serves both cases. ### How it was tested Unit tests in `internal/operator/packagesource_reconciler_test.go` cover the forced rebuild on generation 2 and on a later generation, every case where no rebuild must happen, the rebuild settling without a second one, both failure orders (force fails, recording the generation fails), and a stale recovery counter. Each condition of the new predicate was mutated once to confirm a test fails. I also ran this commit's operator image on an arm64 test cluster running v1.7.0-alpha.3, the one where #4593 was found. Before that I had repaired its bucket generator by hand: deleting the stale ExternalArtifact and requesting a reconcile produced an artifact with `charts/cozy-lib` and both bucket HelmReleases went Ready. With the patched operator, the only generator past generation 1 got exactly one forced rebuild and the verified-generation annotation. source-watcher reported all its ExternalArtifacts up to date. The digests of all 214 ExternalArtifacts and the revisions of all 104 HelmReleases matched a snapshot taken before the swap, and every PackageSource and HelmRelease stayed Ready. The race itself was not reproduced on the cluster, since that needs a spec change landing during a build. Closes #4593 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(operator): rebuild an ArtifactGenerator once after each spec change, so an upgrade no longer leaves a component on artifacts built from the previous spec (for example a chart missing a library it now needs). ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved PackageSource recovery when an artifact generator reports ready: the system now verifies the generated output and triggers a rebuild when needed. * Failed rebuild or verification attempts can be retried during a later reconciliation, and recovery tracking is cleared once the rebuilt generation settles. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
adc2dcb5ca |
fix(operator): rebuild each changed ArtifactGenerator generation once
source-watcher builds an ArtifactGenerator from the spec it read when its reconcile started, but its final status patch re-reads the live object and marks Ready with the live generation. If the spec changes during a build, which an upgrade does as it rolls out a new packages source and new copy rules within seconds of each other, the generator reports the new generation as built while its artifacts come from the old one, and source-watcher never rebuilds them. A chart can then ship without a library its templates now require. The generator status cannot tell a stale build from a good one, so the operator forces one rebuild of every changed generation once it reports Ready, through the existing force-drift path. An unchanged spec rebuilds to the same digest, so nothing downstream moves. The recovery messages no longer name one upstream race, since they now serve two. Upstream: fluxcd/source-watcher#384. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
a09eded9ec |
fix(kubernetes-worker-image): wait for goldens to import before Ready
The Talos worker image catalog renders its goldens the same way the VM image collection does: DataVolumes with immediate bind whose import starts at install. With no health check the release went Ready as soon as helm applied them, so a golden that never imported left the catalog green. Gate it on the DataVolume phase with the same expressions. Worker pools read each golden directly and do not wait on the catalog status, so the gate changes what the status reports and delays no pool. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
51ac18776e |
fix(vm-default-images): wait for golden images to import before Ready
Every image is a DataVolume with immediate bind, so its import starts at install, but nothing waited for it: the HelmRelease went Ready as soon as helm applied the DataVolumes. An image that failed or stalled left the component green, and the failure surfaced later on a vm-disk cloning from it, far from the cause. Gate the release on the DataVolume phase. Only Succeeded counts as ready, and a Failed phase, when CDI sets one, ends the wait at once. Every other phase, and a missing status, keeps the wait open until the install window expires. There is no per-component timeout, so a download longer than that window is retried as an upgrade until it finishes, and the retry strategy never uninstalls the volumes. Signed-off-by: Aleksei Sviridkin <f@lex.la> Assisted-by: LLM |
||
|
|
7a0aab2bdc |
refactor: apply the Go 1.27 modernizers
Run go fix over every first-party module: interface{} becomes any,
hand-written loops become slices, maps, min/max and range-over-int,
errors.As becomes errors.AsType, pointer helpers become new(expr) and
are inlined at their call sites, and so on. Every rewrite keeps
behaviour.
Three rewrites stay out. embedlit flattens embedded-field literals
such as ObjectMeta, syntax only Go 1.27 accepts, which would break
backports to release branches still on Go 1.26. omitzero drops
omitempty from struct fields, and openapi-gen reads omitempty as
optional, so metadata, spec and status would turn required in the
published schemas. And errorsastype rewrote an errors.As whose
target, apierrors.APIStatus, does not implement error, which
errors.AsType rejects at compile time.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
|
8f0f29e1bd |
test(operator): pin how component values files are layered
The first values file overwrites the component's values.yaml and each later one merges into it; no test held that order or the strategies. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
aa20495df3 |
feat(marketplace): register a tapped repository's apps on connect
Connecting a marketplace repository materialized its PackageSource but created no Package, so its ApplicationDefinitions never rendered and its apps never appeared in the dashboard catalog until a manual `cozypkg add` the dashboard cannot drive (the package list is computed from the ApplicationDefinitions, which do not exist yet). The tap materializer now creates a tap-managed registration Package for each materialized PackageSource, the way the platform ships a Package per built-in app, so the ApplicationDefinition registrations deploy on connect and the apps become browsable; the app components carry no install block and are still instantiated per user resource. Privileged install components of an unconfirmed tap auto-registration (a Package that still carries the marketplace-tap label) are refused on several best-effort layers: the Package reconciler skips their HelmRelease and prunes a lingering one at the install site, and the PackageSource reconciler withholds their ExternalArtifact so the helm-controller has no privileged content to deploy, re-running on confirmation via a Package watch. `cozypkg add --allow-privileged` sheds the label as an explicit ownership handover, which is the confirmation; a platform Package never carries the label. A benign component flipped to privileged in a new revision can still be materialized by source-watcher before these layers observe the flip; closing that race fully requires decoupling the catalog registration from install and is tracked in issue #4359. `cozypkg untap` and the dashboard disconnect remove the registration Package (de-registering the apps) under a UID + ResourceVersion precondition, so a concurrent `cozypkg add` handover is never clobbered; running instances are left but unmanaged. `cozypkg add` re-opens a tap auto-registration for variant selection and hands it over. The cozystack-api ClusterRole gains delete on packages for the disconnect path. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
afc003c464 |
feat(postgres): database read-replica autoscaling via KEDA (#3954)
## What this PR does Adds automatic horizontal autoscaling of a PostgreSQL database's **read replicas** in response to load, implementing the merged design proposal [Database Horizontal Autoscaler](https://github.com/cozystack/community/tree/main/design-proposals/database-horizontal-autoscaling) (community#44). The mechanism is entirely stock — no bespoke operator, no new CRD: - **KEDA as an optional platform component** (`packages/system/keda`, wired into `core/platform` as a source + system-bundle entry). It claims the `external.metrics.k8s.io` APIService that nothing served before. - **`cozy-lib.keda.scaledObject` helper** (`packages/library/cozy-lib/templates/_keda.tpl`) renders a KEDA `ScaledObject`, fail-closed on missing/invalid arguments. - **`packages/apps/postgres` autoscaling** — an `autoscaling` values block (`enabled`, `minReplicas`, `maxReplicas`, `metric`, `target`, `maxReplicationLagSeconds`, `dryRun`, `transition`) validated by `values.schema.json`; under active autoscaling the chart renders a `ScaledObject` and seeds `spec.instances` to a constant floor (`max(replicas, effectiveMin)`), while the postgres release is forced to client-side (two-way) Helm apply so the seed is a merge no-op on upgrade and the KEDA-managed HPA — the runtime writer via the CNPG `Cluster` scale subresource — is not reverted. - **Single-value metric (§1)** — one PromQL query returns `Σ(read load over standbys) + target`, so a stock HPA computes `desired = ceil((Σ+target)/target) = 1 + ceil(Σ/target) = primary + desiredRead`, with no per-pod emission. - **The four static brakes (§5)** — quorum floor `max(minReplicas, maxSyncReplicas+1, 2)` (quorum wins over `maxReplicas`), scale-down pacing (`Pods/1/600s`), a replication-lag clamp folded into the query (freeze both directions while lag exceeds the threshold and the primary is writing, with `max_over_time` hysteresis), and dry-run (render paused). - **Two-phase enable/disable migration** via a `transition` sub-flag: phase 1 stages `spec.instances` to the live count and brings KEDA up with its HPA paused in both directions; flipping to phase 2 hands the runtime count to KEDA, and because the seed re-renders the same count the flip is a merge no-op with no dip. - **Observability** — a database-autoscaling Grafana dashboard and PrometheusRule alerts. **Live validation (dev cluster, CNPG 1.30.0).** The full path works end-to-end: the CNPG `Cluster` scale subresource serves `status.selector`, KEDA renders the HPA, the metric reaches it (`ScalingActive=ValidMetricFound`, `TARGETS 151/150`, `desiredReplicas=2 = 1 primary + 1 replica` — the §1 arithmetic), and the lag-clamp query returns the base value on live VictoriaMetrics. Two packaging fixes were needed and are included: KEDA must run on the cozystack `cozy.local` cluster domain (the upstream `cluster.local` default makes the metrics gRPC target NXDOMAIN and every HPA hangs on `FailedGetExternalMetric`), and the operator's client-go throttle is raised so a single-replica operator does not lose its leader lease under scale-subresource polling. Implements #3953. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. Two repositories are affected; both are follow-ups to open once this lands (and after its dependency #3951 merges and the values surface is final), left unticked here for a maintainer to confirm rather than claimed with a speculative link: - **cozystack/terraform-provider-cozystack** — this adds the nested `autoscaling` block to `packages/apps/postgres/values.schema.json`. The provider hand-models each app's fields (schema + model + expand/flatten), so it needs the new block to expose it in Terraform. - **cozystack/website** — KEDA is a new platform component (`guides/platform-stack`), and the postgres reference page gains the `autoscaling` params. The managed-app reference is regenerated from `README.md` by the release bot; the platform-stack component list is hand-maintained. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: to open (new platform component + postgres autoscaling params) - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: to open (new `autoscaling` block in postgres values.schema.json) ### Depends on - #3951 — bump CloudNativePG to 1.30.0. The HPA can only drive the CNPG `Cluster` scale subresource on CNPG ≥ 1.28.4, where it serves `status.selector`; without it the HPA fails with `InvalidSelector`. This PR should not merge before #3951. ### Release note ```release-note feat(postgres): add optional horizontal autoscaling of read replicas via KEDA, configured through the new `autoscaling` values block; KEDA ships as a new optional platform component. Requires CloudNativePG >= 1.28.4 for the Cluster scale subresource selector. Note: every Postgres release now applies client-side (`release.cozystack.io/helm-server-side-apply: "false"`) so the autoscaler's live `/scale` count is not reverted by Helm; a kind-wide side effect is that chart-rendered fields on Postgres releases are no longer re-asserted by server-side-apply drift correction (driftDetection is unset), a documented trade-off with a per-instance apply strategy as follow-up. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable PostgreSQL read-replica autoscaling with KEDA. * Added CPU and read-connection metrics, replica bounds, replication-lag safeguards, dry-run mode, and transition handling. * Added optional KEDA platform support, Grafana autoscaling dashboard, and Prometheus alerts. * **Documentation** * Clarified integer thresholds, prerequisites, configuration, and transition guidance. * **Bug Fixes** * Improved validation and actionable errors for unavailable KEDA or invalid settings. * **Tests** * Expanded coverage for autoscaling, observability, package selection, and validation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
8738063287 |
fix(operator): reject tar entries containing ".." before any path operation
safeJoin already rejected traversal and re-checked the joined path against destDir, but the guard sits behind a call boundary that CodeQL's go/zipslip barrier does not follow, so the extraction loop reported three high-severity arbitrary-write alerts. Reject the entry name up front instead, which is both the fail-closed spelling and the shape the analysis recognises. Extend the traversal test to cover a ".." buried mid-path, not just a leading one. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
1653da967b |
refactor(marketplace): drop artifact-prefix injection and reserved names
Without the community. rename, a tapped repository's cozyrds ApplicationDefinition references its component's artifact by the concrete <packagesource>-<variant>-<component> name again, so the operator no longer injects a cozystackArtifactPrefix value and the -rd chart renders its cozyrds asset with a plain Files.Get. cozypkg init scaffolds the concrete chartRef. Reserved name prefixes are removed: repositories choose their own PackageSource names, and a clash with a core component is caught on-cluster at tap time (the collision check) rather than by refusing prefixes offline. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
95badea1c8 |
refactor(marketplace): identify taps by marker label, drop community rename
Tapped repositories keep their own declared PackageSource names instead of being renamed under the community. prefix. Tap-ness is now marked by the marketplace-tap label (and source annotation) on the OCIRepository and every materialized PackageSource, so untap, the Flux-source cleanup, and idempotent re-tap key on the marker rather than a name prefix. A declared name that clashes with an existing core component (or another tap) is rejected: the CLI tap fails fast before applying anything, and the operator materializer refuses to overwrite the foreign object, records the reason on the source (annotation and Event), and does not stamp the revision so a corrected artifact retries. The shared check lives in internal/marketplace/collision. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
0c0609ad63 |
fix(marketplace): resolve community chartRef via injected artifact prefix
Fixes the rename-vs-chartRef break found on dev3: a community-tapped
PackageSource is renamed to community.<org>.<repo>, so the assembled artifact
name changes, but a repo's cozyrds ApplicationDefinition baked chartRef.name
under the author's original name and the ExternalArtifact it referenced no
longer existed, leaving every community-installed app permanently ArtifactFailed.
Implements @kvaps's option 2 (derive at the -rd component, keep platform charts
literal):
- naming.ArtifactPrefix exposes the per-(source,variant) artifact-name prefix.
- The Package reconciler injects it as the cozystackArtifactPrefix Helm value,
but ONLY for community.-prefixed sources, so platform HelmRelease values are
byte-for-byte unchanged (no churn, no platform chart edits).
- cozypkg init now scaffolds a community-style -rd chart: cozyrd.yaml renders the
cozyrds asset through tpl, and chartRef.name templates
'{{ .Values.cozystackArtifactPrefix }}-<app>', resolving to the real artifact
whatever the tap-time rename produces.
- validate skips the dangling-chartRef check for a templated name (it cannot be
cross-checked at authoring time; the operator resolves it at install).
Verified end-to-end: init -> validate is clean, and helm template with the
prefix value renders chartRef.name to the exact artifact the reconciler emits.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
|
950268cccc |
feat(marketplace): reject reserved PackageSource name prefixes
From @kvaps's review: a repository author could name a PackageSource 'cozystack.<x>' or 'community.<x>' and claim the platform or marketplace namespace. Until now validate only rejected an empty metadata.name and the forced rename in the materializer was the sole guard. - validate now flags 'cozystack.'- and 'community.'-prefixed PackageSource names as an error. The escape hatch --allow-reserved-names lives on the caller side only: the index gate (validate-entry.sh) never passes it, so a submitter cannot hand it to themselves. - cozypkg init refuses the same prefixes early with a neutral-name hint. - Document why the ClusterIP fail-open is safe: it falls back only on address resolution, and integrity stays enforced by the fail-closed digest check, so a wrong host yields a mismatch error rather than a content substitution. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
b96a97e5c4 |
fix(marketplace): resolve tap artifact host to ClusterIP for the hostNetwork operator
The tap materializer fetches the Flux artifact from status.artifact.url, which is a cluster-DNS name (flux.<ns>.svc/...). cozystack-operator runs with hostNetwork: true and dnsPolicy: ClusterFirst, so it resolves against the host's DNS rather than CoreDNS and that name is NXDOMAIN — every dashboard-driven connect (API Create, which relies entirely on the materializer) then hangs in 'connecting' forever, while the CLI tap works only because it pulls via 'flux pull artifact' from the public registry. Resolve the artifact host to the backing Service's ClusterIP (which a hostNetwork pod reaches directly, the same way the in-cluster apiserver is reached via KUBERNETES_SERVICE_HOST) before fetching. This fails open: a non-cluster host, a missing Service, or a headless/empty ClusterIP falls back to the original URL, so clusters where DNS already works are unaffected. Services are readable through the operator's existing cluster-wide get/list/watch grant. Found on dev3 via cozystack-pr-test; the connect->materialize path never worked on a stock operator deployment before this. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
c9accd6086 |
fix(marketplace): address maintainer review — url collisions, fail-closed digest, centralized prefix
From @kvaps's review of the untrusted-artifact path: - Refuse to silently retarget: two repositories that share an org/repo path on different registry hosts derive the same tap name. The API and CLI now reject a connect when an existing source points at a different spec.url, instead of blindly repointing the first at the second. - Fail closed on integrity: verifyAndExtract now errors on an absent digest (not just a mismatch), and the reconciler waits for status.artifact.digest, so a tap artifact is never materialized without a verified digest. - Centralize the load-bearing community. prefix in internal/marketplace/tapconst (removing two duplicate consts and several literals across the CLI, API, materializer, and dashboard) so the disconnect guard, the community badge, and the anti-shadowing guarantee cannot drift. - Split multi-document YAML with the apimachinery reader so a PackageSource whose values contain a separator-looking line is not fractured. (The short-name version-pin drop was already fixed in an earlier commit.) Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
3ce8279019 |
fix(marketplace): prune stale taps, preserve source state, scope RBAC, pull by digest
Address the remaining review findings: - The materializer now prunes PackageSources it materialized from an earlier revision that the current artifact no longer contains (including a single/multi rename), so a removed package leaves the catalog; installed Packages stay. - A repeat connect updates the fetched OCIRepository's spec, label and name annotation instead of replacing the object, so the operator's finalizer and materialized-revision annotation survive; covered by a test. - OCIRepository write verbs move from the cluster-wide cozystack-api ClusterRole to a namespaced Role in cozy-system (get/list/watch stay cluster-wide for the watch cache), since the API only manages tap sources there. - 'cozypkg validate --require-signature' now pulls the exact manifest digest cosign verified, closing the tag-mutation window between verify and pull. - safeJoin rejects any '..' path segment outright (defense in depth and to satisfy the Zip-Slip scanner) on top of the existing containment check. - Document why the index gate's merge/label steps need a workflow_run split for fork pull requests. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
7b703f2859 |
fix(cozypkg): satisfy golangci-lint and derive a valid kind for hyphenated apps
Address the linters flagged in review: check the previously-ignored writer, Close, and RemoveAll return values (errcheck), lowercase an error string (ST1005), drop the embedded-field selector (QF1008), remove an unused const, and switch the materializer's post-finalizer requeue from the deprecated Result.Requeue to RequeueAfter. Also fix 'cozypkg init': capitalize now builds an upper CamelCase kind from a hyphenated --app (foo-bar -> FooBar) so the scaffolded ApplicationDefinition is not rejected by the apiserver at install time; covered by a test. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
3643fceceb |
refactor(marketplace): fail-safe re-tap rollback and single-source artifact naming
Close the two non-blocking notes from the third review round: - objectExists now fails safe: it returns 'not existing' (eligible for rollback) only on a definite NotFound; any other Get error (RBAC, throttling, timeout) is treated as possibly-pre-existing, so an idempotent re-tap that hits a transient probe error cannot delete a pre-existing resource on rollback. Mirrors the fail-direction of sourceStillReferenced; covered by a new interceptor-based test. - The PackageSource reconciler now calls internal/marketplace/naming.ArtifactName instead of inlining the formula, so the artifact-name convention has a single definition shared by the producer, the marketplace backend, and the validator, and the parity test guards producer-vs-helper. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
e61d2c7b53 |
feat(marketplace): connect a community tap from the API with async materialization
Add server-side connect to complete the marketplace write path. Creating a Tap records the intent only: the API validates the oci:// reference and creates a labeled Flux OCIRepository (with an optional pull-credential secretRef), never blocking the request on a registry pull, and returns a Tap whose status shows materialization pending. The connect inputs (url, tag, secretRef) are write-only and are never echoed back or returned on read. A new tap materializer reconciler in the operator watches community-tap OCIRepositories, and once source-controller has pulled the artifact, fetches the tarball over the Flux artifact URL (digest-verified, size-bounded, path-traversal guarded), parses the PackageSource(s) it carries, and materializes them under the community. prefix with their sourceRef repointed at the tap's source. A finalizer cleans up the materialized PackageSources when the source is removed, leaving installed Packages in place. The shared label/annotation keys live in internal/marketplace/tapconst so the API and the operator cannot drift on the connect-to-materialize handoff. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
27d7dbb140 |
feat(api): add per-app helm apply-strategy override annotation
Add release.cozystack.io/helm-server-side-apply on the ApplicationDefinition, threaded through ReleaseConfig into every emitted HelmRelease's Install/Upgrade.ServerSideApply. Empty leaves the helm-controller default (server-side apply on v1.5.0); "false"/"disabled" forces client-side apply. The postgres read-replica autoscaler needs this: KEDA's HPA writes the CNPG Cluster's /scale subresource (spec.instances) while the chart renders a constant seed. Under helm-controller's default server-side apply the rendered field is force-owned and reverts KEDA's live value on every reconcile. Under client-side apply helm-controller patches the CRD from the previous-vs-new rendered manifest without consulting live state, so the unchanged constant yields no patch and the /scale value survives (verified against Helm v4 createPatch: unstructured/CRD objects take the two-way jsonpatch.CreateMergePatch(original, modified) path unless threeWayMergeForUnstructured is set, which helm-controller v1.5.0 leaves at its default). Guarded by TestConvertApplicationToHelmRelease_ServerSideApplyOverride on the tenant-release builder (convertApplicationToHelmRelease), and by TestParseHelmServerSideApplyAnnotation. Removes a stale package_reconciler_test guard that asserted a false invariant on the wrong (platform Package) builder. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
826f08972f |
fix(apps/postgres): make autoscaler fail-safe, dip-free, and observable
Address review findings on the KEDA read-replica autoscaler where the
implementation diverged from the merged design proposal.
Fail-safe read load. Gate the `or vector(0)` floor on the existence of ANY CNPG
instance pod (`count(kube_pod_labels{...instance_role=~".+"}) > 0`) — primary or
replica — so the query is empty ONLY when no CNPG pod labels exist at all (a
broken kube-state-metrics pipeline), not during initial provisioning when the
sole replica-side pod is CNPG's join Job (jobRole, not instanceRole). Also pin
`ignoreNullValues: "false"` on the trigger: KEDA's prometheus scaler defaults it
true, turning an EMPTY result into metric 0 (not an error) — the HPA would then
desire 0 and clamp to the floor, i.e. silently scale a broken-pipeline cluster
DOWN. With it false an empty result is a scaler error, so the HPA HOLDS and
DatabaseAutoscalerScalerErrors fires; gating on any instance role keeps a healthy
cluster (including a long basebackup) from tripping that alert.
Constant seed = max(replicas, effectiveMin), so enable/dryRun/transition flips
are a merge no-op with no dip and never undercut the quorum floor.
Fail closed when the quorum floor exceeds maxReplicas (was: effectiveMax silently
raised both bounds, unbounded for a large maxSyncReplicas).
Dry-run / transition keep the HPA (unidirectional pause pair, not the HPA-deleting
full pause) but both SelectPolicy=Disabled, so desired normalizes to current — the
recommendation is read from the read-load metric, not a diverging desired count.
Observability. DatabaseAutoscalerAtMax keys on desired >= maxReplicas, NOT the
ScalingLimited condition (which is also true at the floor, so it fired on every
idle cluster with wrong "raise maxReplicas" advice). Dashboard read-load panels
are per-cluster (label_cnpg_io_cluster) with a CPU panel for ReadCPUUtilization
and the exact clamp in descriptions; ScaleStuck's Pending intersection is scoped
to CNPG pods; alerts.yaml narrative comments moved to Go-template comments so they
do not ship into the PrometheusRule.
Guard the seeding invariant: TestBuildHelmReleaseSpecNoForceOrDriftDetection.
Regenerated README/schema/types.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
96a9652fb1 |
feat(api): typed waitStrategy/healthCheckExprs for kstatus readiness
Add first-class fields so a package/app can declare the Flux helm-controller v1.5.0 kstatus health gate for the custom resource it wraps, so HelmRelease.Ready means "the CR is actually healthy" instead of "helm applied the CR" (#2642). API-owner approved in #3263. - api: WaitStrategy (scalar, maps to HelmReleaseSpec.WaitStrategy.Name — a deliberate simplification) + HealthCheckExprs ([]kustomize.CustomHealthCheck, the upstream type verbatim) on both ApplicationDefinition.spec.release (the apiserver path) and ComponentInstall (the operator path). - config.ResolveWaitStrategy: shared by both HelmRelease builders. Couples the default — healthCheckExprs imply the poller strategy when none is set, since exprs are only evaluated under poller. This removes the "must also set waitStrategy: poller" footgun and does not depend on the controller's global default. - thread through buildHelmReleaseSpec (operator) and convertApplicationToHelmRelease (apiserver); the apiserver loads the typed spec.release fields in start.go. - regenerate deepcopy, the ApplicationDefinition and PackageSource CRDs, and openapi. The backupstrategy CRDs also regenerate — benign k8s-0.35 PodSpec description drift, not a functional change. disableWait is intentionally not added (YAGNI): the only known need (the kubernetes tenant-addon deadlock) was resolved child-side, and waitStrategy: legacy is not equivalent (legacy still waits Helm-v3-style; disableWait skips waiting entirely). Add it later when a concrete leaf needs a true no-gate escape hatch. Refs #2642 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
0d98e27981 |
fix(operator): hold final backoff window before give-up, note upstream version skew
Address lexfrei's non-blocking follow-ups on the LGTM approval: Follow-up #1 — give-up preempted the final backoff. `decideRecovery` returned `recoveryActionGiveUp` as soon as `attempts >= maxRecoveryAttempts`, without checking whether the final-force backoff window had elapsed. The Owns()-watch re-fire from the N-th force-drift's own writes triggered a reconcile milliseconds after the force, so the effective time-to-give-up was ~7.5m (30s+60s+2m+4m) — not the ~11.5m documented on the constants block. The final 4-minute response window the last nudge was supposed to grant source-watcher was never actually granted. Fix: extend the exhaustion branch to hold `Wait` until `now - lastRecoveryAt >= backoffFor(maxRecoveryAttempts)` before switching to GiveUp. Now the documented ~11.5m budget matches the observed behaviour, and the last force gets a real chance to succeed (Owns watch → source-watcher Ready=True copy-through) before we surface SourceWatcherStalled. Tests: * `TestDecideRecovery` gains a case pinning `Wait` when attempts=maxRecoveryAttempts but the final backoff has not yet elapsed (the exact re-fire shape the previous behaviour tripped on). * `TestUpdateStatus_OwnMarkerAttemptsExhaustedButFinalBackoffPending` drives the full updateStatus path against the same shape, asserting the PackageSource stays in Unknown/AwaitingSourceWatcherRecovery and the RequeueAfter equals the remaining backoff, NOT zero (GiveUp). Follow-up #3 — upstream version-skew note. The Go module dependency tracks `fluxcd/source-watcher/api/v2 v2.0.3` (CRD types package only), but the deployed binary this workaround targets is pinned to v2.1.0 at `internal/fluxinstall/manifests/fluxcd.yaml`. Comments referencing `drift.go:52` / `controller.go:164/254` describe the deployed binary, not the compiled API package. Documented on the constants block so a future flux bump re-verifies the anchors AND the watch predicate / drift-detection heuristics, which are the load-bearing upstream assumptions. Follow-up #2 (envtest over full Reconcile→Owns→updateStatus cycle) is architectural and out of scope for this PR — noted for a follow-up. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
|
37421d2fec |
docs(operator): honest JSON-merge-patch tradeoff, clarify asymmetric OG check, warn on partial force-drift
Self-review nits raised in a skeptical pass over the current tip: * The status-patch docstring in forceArtifactGeneratorDrift described the JSON-merge-patch full-array-replace on `status.conditions[]` as "net-neutral". That was too optimistic: if source-watcher writes a concurrent Reconciling condition between our Get and our Patch, our patch does transiently wipe it. source-watcher restores it on its next reconcile (which our requestedAt bump is about to trigger), but downstream tooling watching for Reconciling can observe the momentary absence. Rewrite the comment to name that trade-off explicitly and point at server-side apply as the fuller fix if the operator ever migrates to SSA globally. * The unified own-marker/progressing/stuck gate in updateStatus is asymmetric on ObservedGeneration — isProgressingInRecovery checks it, isOwnMarker does not. That asymmetry is deliberate (a stale own-marker after a spec edit must still route to the state machine to avoid leaking the synthetic reason to the PackageSource — exactly the B2 regression), not an oversight. Add a comment explaining the reasoning so a future edit doesn't "symmetrise" the check and re-open B2. * forceArtifactGeneratorDrift's step-2 (metadata patch) failure was silent when it happened after a successful step-1 status patch. In that state source-watcher has been nudged, but the tracking annotations never persisted, so the next reconcile reads attempts=0 and re-forces from scratch instead of accumulating toward SourceWatcherStalled — a persistent metadata-write-only apiserver malfunction would loop forever. Log the failure loudly with the intended attempt count so operators can spot this specific mode from cozystack-operator logs. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
596427deb3 |
docs(operator): scrub stale awaitSourceWatcherResponse references from comments and test names
Self-review after the previous commit removed the awaitSourceWatcherResponse helper (routing unified through maybeRecoverArtifactGenerator's state machine via a single own-marker / progressing-in-recovery / stuck gate). Four comments and two test names still referenced the now-deleted symbol — misleading anyone tracing the architecture. * maybeRecoverArtifactGenerator docstring: rewrote the trade-off paragraph to describe the unified gate and Wait branch instead of the removed helper. * TestMaybeRecoverArtifactGenerator_DoesNotResetCounterOnFreshTransition comment: same rewrite. * TestUpdateStatus_OwnMarkerRoutesToAwait renamed to TestUpdateStatus_OwnMarkerHoldsTrackingViaStateMachine + comment rewritten to describe the state-machine Wait branch. * TestUpdateStatus_ProgressingDuringRecoveryRoutesToAwait renamed to TestUpdateStatus_ProgressingDuringRecoveryHoldsTracking + comment rewritten in parallel. Pure doc / naming; no code or test-body changes. Tests still pass with -race, vet clean, golangci-lint 0 issues. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
7b08fcdca9 |
fix(operator): unify recovery gating through state machine, close source-watcher-unresponsive dormancy loop
Self-review on 467b788b8 caught a MAJOR bug introduced by the separate awaitSourceWatcherResponse helper: if source-watcher never responds to a forced drift (network partition, missing predicate, dead pod), our own reasonRecoveryForced marker keeps reflecting back via Owns() forever, and the wait-forever helper schedules a 1-second requeue tight loop without ever advancing the retry state machine. SourceWatcherStalled give-up was unreachable. Fix: remove awaitSourceWatcherResponse and unify all three recovery entry conditions — our own marker, source-watcher's Progressing during rebuild, and the classic first-time stuck detection — through a single gate that routes to maybeRecoverArtifactGenerator. The state machine's decideRecovery is the single arbiter of wait/force/give-up, and its Wait branch is bounded by the same backoff schedule as Force. When backoff elapses on a still-reflecting own-marker the machine escalates to another force (bumps attempts, re-stamps requestedAt); when attempts exhaust it surfaces SourceWatcherStalled honestly. The own-marker filter also no longer requires attempts>0, so an operator crash between step 1 (status patch) and step 2 (metadata patch) of forceArtifactGeneratorDrift is still routed through the state machine on the next reconcile rather than leaking the synthetic marker onto the PackageSource. Also address the non-blocking notes from the same self-review: * Docstring on maybeRecoverArtifactGenerator was still describing the removed reset-counter branch and the "requestedAt alone is a no-op" framing. Rewrite to explain the current shape: status write triggers the drift signal, requestedAt bump triggers the reconcile event — both are required, neither alone works. * TestForceArtifactGeneratorDrift_MetadataPatchFailure_RollbackAnnotations now asserts reconcile.fluxcd.io/requestedAt is also rolled back on the metadata-patch failure branch (symmetric with the tracking annotations). * TestUpdateStatus_OwnMarkerRoutesToAwait asserts an exact 25s RequeueAfter (backoffFor(1)=30s minus 5s elapsed) so a silent regression in the backoff arithmetic is caught. New regression tests for the source-watcher-unresponsive path: * TestUpdateStatus_OwnMarkerStale_TriggersRetry: with attempts=1 and lastRecoveryAt older than backoffFor(1), the reconcile must escalate through the Force branch and bump attempts to 2 rather than idle in a tight requeue loop. * TestUpdateStatus_OwnMarkerAttemptsExhausted_GivesUp: with attempts=maxRecoveryAttempts and the marker still reflecting back, the reconcile must set PackageSource Ready=False / SourceWatcherStalled instead of extending the loop indefinitely. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |