The pre-delete hook read a release's state once, so a single failed read was taken as the release being handled: its fence came off before helm-controller's uninstall, and the report counted it as released cleanly. A state read is now retried within the release's share of the window, a release whose state still cannot be read keeps its fence for the wait like one helm-controller has not handled, and the end of the wait reads each again before naming what the release was left to. A release still suspended after the sweep is named as suspended but not by flux-plunger, which also holds when no manager is recorded for the field at all. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
flux-plunger
Recovers a HelmRelease stuck with has no deployed releases: it suspends the release, records the revision it is about to reclaim in the flux-plunger.cozystack.io/last-processed-version annotation, deletes that stale Helm storage Secret, and unsuspends. The annotation goes first so that a write which fails never lets a later pass delete the next revision; a release whose annotation names a revision whose Secret is still there is one whose delete failed and will be retried.
The uninstall fence
While a release is suspended for recovery it also carries the finalizer flux-plunger.cozystack.io/uninstall-fence. helm-controller skips the Helm uninstall of a suspended HelmRelease that is deleted, so without the fence a delete arriving during recovery would leave the release's resources behind. With it, flux-plunger lifts its suspension, helm-controller runs the uninstall, and flux-plunger then removes the fence. A delete held this way for more than ten minutes raises an UninstallFenceHeld Warning on the HelmRelease.
When flux-plunger stops it gives up the fences it holds itself, within its termination grace period (shutdownGracePeriodSeconds, 90 by default, accepted between 40 and 140): its manager is given 10 seconds to stop, and the drain 25 seconds less than the grace period. The next flux-plunger starts while the old one is still terminating, on a rollout as on an eviction or a node drain, so a single writer is kept by leader election: the old pod holds its Lease through the drain without giving it up, and the Lease outlasts the drain, so a new pod waits for it to run out, a little under two minutes at the default, before it acts. A recovery in progress is aborted: the suspension and the fence go together, and the next flux-plunger starts the recovery again. A release being deleted has flux-plunger's suspension lifted and keeps the fence until helm-controller has run its uninstall, which the drain waits for; one helm-controller has not handled when the grace period runs out keeps its fence. Uninstalling this chart then releases whatever is left: a pre-delete hook stops flux-plunger, waits out its grace period, and lifts any remaining suspension and fence. It lifts a suspension only where flux-plunger is the sole manager holding it, which is the outcome flux-plunger's own relinquish has, so a suspension another manager also holds stays; it does so with a patch of spec.suspend alone rather than a kubectl apply, which on a release last written by a client-side apply would take over and strip the fields that apply set. A release being deleted that only the fence still holds keeps it while helm-controller, which sees the generation the lifted suspension wrote, runs the uninstall, until a little before the hook's retry window closes; one helm-controller has not handled by then is deleted without an uninstall, and so is one whose suspension another manager holds, since helm-controller skips the uninstall of a suspended release, and the report names each as such. The Job's 240-second deadline sits under the 300 seconds helm-controller allows a pre-delete hook by default. A Deployment that cannot be read or scaled down (flux-plunger would keep running and could fence behind the sweep) ends the hook before any fence is touched, a listing that never succeeds ends it before any fence is touched too, and a fence the hook cannot remove ends it after the sweep; either way the hook exits non-zero, which fails the uninstall and makes helm-controller retry it, hook included. A flux-plunger Deployment that is already gone has nothing to stop, and a flux-plunger pod that is still there after its grace period, typically one stuck Terminating on a NotReady node, is named and the sweep goes on, since failing on it would fail every retry until the node returns or the pod is force-deleted; because such a pod may still be fencing, the fences are listed again after the sweep, and any found fails the hook. The hook stops, waits for and lists again only for the flux-plunger of this release, found by its release's labels in the release's namespace. Another one, a second installation under another release name, a pod whose labels were changed, or a binary run outside the cluster, is not stopped by the hook: it keeps the fences it sets and gives them up itself when it stops, and a fence it sets after the hook's listing is its own to release. Its report, released N of M fenced HelmReleases, K without an uninstall and a line per release, stays readable for ten minutes after the Job finishes, success included: kubectl logs --namespace cozy-fluxcd job/flux-plunger-uninstall-fence-cleanup.
If flux-plunger is merely stopped (scaled to zero) rather than uninstalled, or killed without a graceful stop, a fenced release that is deleted and that helm-controller has not uninstalled stays Terminating until flux-plunger runs again. To release one by hand, after confirming its uninstall has run or is not wanted, remove the finalizer at the position INDEX it has in metadata.finalizers; the test operation makes the patch fail rather than remove another finalizer if the list has changed:
kubectl patch helmrelease NAME --namespace NAMESPACE --type json --patch '[{"op":"test","path":"/metadata/finalizers/INDEX","value":"flux-plunger.cozystack.io/uninstall-fence"},{"op":"remove","path":"/metadata/finalizers/INDEX"}]'
Fenced releases can be listed with:
kubectl get helmreleases.helm.toolkit.fluxcd.io --all-namespaces --output go-template='{{range .items}}{{$ns := .metadata.namespace}}{{$name := .metadata.name}}{{range .metadata.finalizers}}{{if eq . "flux-plunger.cozystack.io/uninstall-fence"}}{{$ns}}/{{$name}}{{"\n"}}{{end}}{{end}}{{end}}'
A restore during a recovery
A backup restore that starts while flux-plunger holds its own suspension of the release does not take spec.suspend over: the etcd, CNPG and Velero restore steps write the field only when its value differs, and it is already true. flux-plunger still owns the suspension, so when its recovery ends it sets spec.suspend back to false under the running restore, and helm-controller reconciles the release mid-restore. The window is the few API calls between flux-plunger acquiring the suspension and relinquishing it. If a restore misbehaves right after a recovery, suspend the release again by hand for the rest of the restore.
Suspensions left by an earlier flux-plunger
Earlier flux-plunger releases set spec.suspend through the shared flux-client-side-apply manager, so a suspension one of them left behind cannot be told apart from a migration's or an operator's and is treated as external: it is not cleared automatically. Such a leftover exists only if the old pod stopped in the middle of a recovery, and since helm-controller rewrites no condition of a suspended release, it stays frozen in the failure recovery started from: Ready=False with has no deployed releases. flux-plunger raises a SuspensionNotOwned Warning on a release in that state, and a Normal event on any other suspension it does not own; the processed-version annotation is no sign of a leftover, since it stays on every release flux-plunger ever recovered. To list candidates, suspended releases in that state whose spec.suspend is not owned by flux-plunger's apply:
kubectl get helmreleases.helm.toolkit.fluxcd.io --all-namespaces --output json | jq --raw-output '.items[] | select(.spec.suspend == true) | select([.status.conditions[]? | select(.type == "Ready" and .status == "False" and (.message // "" | contains("has no deployed releases")))] | length > 0) | select([.metadata.managedFields[]? | select(.manager == "flux-plunger" and .operation == "Apply" and .fieldsV1["f:spec"]["f:suspend"] != null)] | length == 0) | "\(.metadata.namespace)/\(.metadata.name)"'
Clearing one is an operator action: confirm nothing else (a migration, a restore) is holding it, then resume it with flux resume helmrelease NAME --namespace NAMESPACE.