6 Commits

Include renames

Author	SHA1	Message	Date
Alexey Artamonov	699555c7cb	fix(fluxcd): give each release its own share of the hook's retries ... The pre-delete hook's retry budget was one window spent in listing order, so a release that could never be read or unfenced, listed first, took all of it and every release after it got a single attempt. The same transient failure then released a fence or not depending on where its release sorted. Each release now gets its own share of the window, capped at the window's end, and the steps that cover every release keep the whole of it. Two comments said more than holds. The retry window is measured from the script's start, not the Job's, so scheduling and the image pull come out of its one-minute reserve rather than being covered by it. And the five-minute conflict retry inside kubectl scale never runs here, since without a precondition it sends a merge patch with no resourceVersion to conflict on. helm-controller's finalizer, which the hook reads to tell whether a release being deleted is still held, is now tied to the constant the controller uses, as the fence already was. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>	2026-10-13 15:09:00 +03:00
Alexey Artamonov	7b84919db3	fix(fluxcd): warn about a foreign suspension only on a stuck release ... flux-plunger raised its SuspensionNotOwned and TerminatingWhileSuspended Warnings on a suspension it does not own when the release carried its processed-version annotation. That annotation is never removed, so every later suspension of such a release, by a migration, a backup restore or an operator, drew a Warning: one telling the reader to resume it, which mid-restore is the one thing nobody wants, and one calling a migration's deliberate suspend-then-delete an accident. Both also missed the leftover they are for when the first recovery stopped before writing the annotation. A suspension an earlier flux-plunger left mid-recovery freezes the release in the failure recovery started from, since helm-controller rewrites no condition of a suspended release. Both Warnings now key on that state instead, and the README's candidate query with them. Two comments are qualified where they claimed more than holds: a stop that overran the manager's grace period can still let a reconcile set a fence after the drain's last listing, and a process stopped before controller-runtime closes Elected holds the Lease yet does not drain. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>	2026-10-13 15:09:00 +03:00
Alexey Artamonov	caeaa6b5e5	fix(fluxcd): make flux-plunger's shutdown drain report what it left ... The drain's only report was an Info line, so an alert on errors never saw fences left behind; it named no release, and printed held: -1 when no listing had answered. A flux-plunger that never led logged that it had released every fence next to the line saying it drained nothing. Fences left behind are now an error naming each release, an unanswered listing says so, and a process that never led says only that. The drain writes its Events itself, since the manager's recorder stops with the manager, but it created one Event object per pass for a release it revisits every second, and each write had a deadline of its own outside the drain's. Repeats now fold into one counted Event, as the manager's recorder folds them, and every write is bounded by the drain. Tests now hold the outcome line, the retry of a release whose fence cannot be given up yet, and the pause between passes, none of which any suite noticed when removed. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>	2026-10-13 15:09:00 +03:00
Alexey Artamonov	8bcb709ce8	fix(fluxcd): refuse a flux-plunger drain deadline that leaves no time ... A drain deadline of zero or less is already spent when the drain starts, so flux-plunger would exit holding every uninstall fence without listing one. The chart could produce one: its 40-second floor was a literal, so raising the manager's stop bound or the margin before SIGKILL, which the drain subtraction is already tied to, would leave the floor rendering a non-positive deadline. The binary now refuses such a flag at startup, before it reaches for a cluster, and the chart's floor and the timings the README and the values comment state are held to the same constants. A whole number of a million seconds or more printed in exponent form and was refused as a fraction; the range is now checked first, so it gets the range message. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>	2026-10-13 15:09:00 +03:00
Alexey Artamonov	42a27567ce	fix(fluxcd): keep Helm uninstall working during flux-plunger recovery ... flux-plunger recovers a "has no deployed releases" HelmRelease by briefly suspending it, deleting the stale storage Secret and unsuspending. helm-controller v1.5.x skips the uninstall and drops its finalizer when it handles a delete on a suspended HelmRelease, and it wakes on the same delete event flux-plunger does, so a delete inside that window orphaned the release's DataVolume, PVCs, RBAC and storage Secret. Lifting the suspension on the delete event cannot win that race reliably. Hold the release with a flux-plunger finalizer while it is suspended. The finalizer is applied together with the suspension in one server-side apply. On a delete flux-plunger lifts its suspension and keeps the finalizer, helm-controller reconciles the unsuspended generation and runs the uninstall, and flux-plunger removes the finalizer once helm-controller's own is gone and that generation is observed. This holds whichever controller handles the delete first. A completed recovery lifts the suspension and the finalizer together, and a terminating release never begins recovery. Decide ownership of a suspension by which field manager owns spec.suspend in metadata.managedFields rather than by an annotation. The apiserver transfers the field away the moment an operator, cozyhr resume or a backup controller writes a different value to it, so a later suspension is never misread as flux-plunger's, and one set through the shared flux-client-side-apply manager (platform migrations, the tenant delete hook, a manual kubectl apply) is never claimed. A release deleted under another manager's suspension is left to that manager's decision. Acquire applies without forcing first and, on a conflict, re-fetches and forces only if spec.suspend is still false: a backup controller owns the field at false through a dynamic-client Update, which a non-forced apply can never take, while true is an active suspension to leave alone. Every apply is preconditioned on the resourceVersion and uid of the read that decided on it. Without the resourceVersion, the unforced apply of true over a foreign true is accepted as shared ownership, a window the optimistic-locked merge patch this replaces had closed, and lifting the suspension and the finalizer from a release read as live would delete it without an uninstall if helm-controller had meanwhile handled a delete. Without the uid, an apply to a release that has gone would create an empty one. The apply carries only spec.suspend and the finalizer, so it cannot leak the required spec.interval. Read the release, every re-check and the storage Secret list from the apiserver rather than the informer cache. A lagging cache would let the processed-version guard delete the revision below the one just reclaimed, and the ClusterRole grants secrets get and list but not the watch an informer needs. Emit a Warning when flux-plunger holds a suspension it failed to clear, and Normal events when it leaves an external suspension alone or defers a recovery. An external suspension on a release flux-plunger has recovered before is a Warning instead: it is most likely one an earlier flux-plunger left under the shared manager, which nothing lifts unless an operator does. A release deleted under another manager's suspension is a Normal event when that was a migration's choice, and a Warning when the release carries flux-plunger's processed-version annotation, since then the skipped uninstall is an accident. Keep a single writer with leader election. A second process able to act under the same field manager could clear a suspension the first is still recovering, and no rollout strategy prevents one: the next pod starts as soon as the old one is terminating, and an eviction or a node drain replaces a pod the same way. The old pod keeps its Lease, unreleased, through its stop and drain, and the Lease outlasts both, so the next pod acts only after the old one is done; the price is that a new pod waits for the Lease to run out. The strategy sets maxSurge 0 with maxUnavailable 1 and stays RollingUpdate because existing installs carry the apiserver-defaulted rollingUpdate block and helm-controller applies server-side, where type Recreate over it is rejected on upgrade. Give the finalizer a way out when flux-plunger itself goes. Once the manager has stopped, within a bounded time, and with it every reconcile, flux-plunger drains its finalizers under a deadline that, with the stop, fits inside the pod's termination grace period. A stop that overran its bound is drained too and still reported, and only a pod that led drains. The drain writes its events straight to the apiserver, since the manager's recorder stops with the manager. A recovery in progress is aborted, the suspension and the finalizer lifted together for the next flux-plunger to start again, and a release being deleted has the suspension lifted and keeps the finalizer until helm-controller has run the uninstall. A release helm-controller does not handle in time keeps it. The waiting lives in the controller, where the envtest suites hold it against a real apiserver's generations and finalizers, rather than in a shell script against a model of them. A pre-delete hook on the chart is the backstop for what the drain could not finish, so uninstalling the component never leaves a release that tenant teardown waits on forever. It stops flux-plunger, waits out its grace period, then lifts any remaining suspension and finalizer without waiting again, naming each release it lets go of before helm-controller uninstalled it. It lifts a suspension only where flux-plunger alone holds it, with a JSON patch preconditioned on the resourceVersion and uid it read: kubectl apply --server-side on a release last written by a client-side apply moves that apply's fields to the applying manager and strips every one the body leaves out. The Job's deadline sits under the 300s helm-controller gives a pre-delete hook by default. A Deployment that cannot be stopped or a listing that never succeeds ends the hook before any finalizer is touched, and a finalizer it cannot remove ends it after the sweep; either way it exits non-zero: a failed pre-delete hook leaves the release untouched and helm-controller retries the uninstall, hook included, from a state it can still converge from, where exiting 0 would remove the component and strand the finalizer. A flux-plunger pod still present after its grace period, typically one stuck Terminating on a NotReady node, is only reported, since failing on it would fail every retry until the node returns; because it may still be setting finalizers, they are listed again after the sweep and any found fails the hook. The Job runs once, since helm-controller's retry is the one that starts afresh, and is kept after success so that its report stays readable; helm.sh/hook-output-log-policy would make helm-controller call a nil output writer (fluxcd/helm-controller#1562). A delete still held after ten minutes raises a Warning, and the held state is re-checked on a timer instead of only on helm-controller's next status write. The chart README documents the finalizer and how to clear it. Run the envtest suites on every pull request through a new make target that fetches a kube-apiserver and etcd with a pinned setup-envtest and takes the HelmRelease CRD from the in-tree Flux manifests: the unit tests drive a fake client, and everything the finalizer rests on (apply preconditions, the finalizer set-merge, real managedFields) only holds against a real apiserver. CI caches those binaries, and the target uses them without asking the release index, so a pull request does not depend on their sources being reachable; the control plane gets a longer start timeout because it starts beside the other test goals. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>	2026-09-15 23:34:52 +03:00
Andrei KvapilandClaude	2d1c8aae02	feat(platform): add flux-plunger controller ... Add flux-plunger controller to automatically fix HelmRelease resources with "has no deployed releases" error. The controller watches HelmRelease resources and performs the following: - Detects HelmRelease with "has no deployed releases" error - Suspends the HelmRelease with flux-client-side-apply field manager - Deletes the latest Helm release secret - Updates annotation with processed version to prevent recursive deletion - Unsuspends the HelmRelease to allow Flux to retry Special handling for suspended HelmRelease: - If suspend=true and latest+1==processed: removes suspend - Otherwise: skips processing as suspended by external process Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com>	2026-01-19 13:20:08 +01:00