The shell translator rewrites test bodies and loses Bats reporting.
The unit lane needs complete TAP results and JUnit reports without
changing how live-cluster suites run.
Sourced production helpers still need a dash pass, and unit test bodies
need nounset without exporting shell options into subprocesses.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
kubectl reads --timeout=0 as "wait forever" and substitutes a week,
so the two `kubectl delete helmreleases --wait=true` calls in the
tenant pre-delete hook hang for as long as any child HelmRelease
cannot finalize. helm returns from Uninstall.Run before deleting
anything, so the release stays in storage and flux retries the same
teardown; helm.sh/hook-delete-policy carries no hook-failed, so the
waiting Job survives between attempts as well.
Bound each wave at 90s. Neither is guarded: nothing stands behind
these deletes, so a wave that did not finish ends the run rather than
report a cleanup that did not happen, and ending it keeps the tenant
modules from being torn out from under applications that are still
deleting. Whatever was deleted stays deleted, so the next attempt
resumes from there.
The bound has a price and it is not zero. A nested Tenant, or an app
of kind Kubernetes whose own hook spends two 120s waits first, misses
90s and converges across uninstall retries instead; neither fits the
300s budget as it stands either. Between those and the fast case sits
a band that does pay: a wave inherits its slowest child, so one that
settles around 95s completes inside the budget today and expires here
instead, trading a clean attempt for one more retry. An unbounded
wave does not fail, it spends the whole budget and reports a timeout
naming nothing, so the trade is worth taking.
Give the Job an activeDeadlineSeconds of 270. It bounds what the
waits do not — the DELETE calls carry kubectl's default
--request-timeout of 0 — and sits under the 300s uninstall budget
flux applies when the HelmRelease carries no timeout of its own,
because the deadline runs from the Job's startTime while that budget
runs from just before the Job is created. At equal values helm gives
up first, on a Job still running.
Cap the pod's termination at 10s, which is part of that arithmetic
rather than a detail of it. Since 1.31 the Job controller withholds
the terminal condition until every pod has finished terminating, so
helm learns of the failure at the deadline plus this period; the 30s
default would land it at 270+30, level with the budget the deadline
is set under. The period is spent in full: the SIGTERM goes to PID 1,
which is a shell running a multi-command script, and a default
disposition sent to PID 1 is discarded by the kernel with no handler
installed to catch it. kubectl is a child and is never signalled.
Set backoffLimit to 0 and restartPolicy to Never. The limit is needed
under either policy, since the default of 6 keeps the failure from
reaching helm inside the budget. The policy matters because under
OnFailure the pod survives its own failure and the kubelet re-runs
the container in place, with the controller applying the cap only
afterwards by summing restartCount: the script re-enters the wave
that just expired and is cut down mid-run. Never leaves a failed pod
the controller sees directly.
Add resource requests to match the sibling hook in
packages/apps/kubernetes. Without them the pod is BestEffort and
first out under node pressure, which a retry inside the Job used to
absorb and a single attempt does not.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>