Aleksei Sviridkin
cf85716444
kubectl reads --timeout=0 as "wait forever" and substitutes a week, so the two `kubectl delete helmreleases --wait=true` calls in the tenant pre-delete hook hang for as long as any child HelmRelease cannot finalize. helm returns from Uninstall.Run before deleting anything, so the release stays in storage and flux retries the same teardown; helm.sh/hook-delete-policy carries no hook-failed, so the waiting Job survives between attempts as well. Bound each wave at 90s. Neither is guarded: nothing stands behind these deletes, so a wave that did not finish ends the run rather than report a cleanup that did not happen, and ending it keeps the tenant modules from being torn out from under applications that are still deleting. Whatever was deleted stays deleted, so the next attempt resumes from there. The bound has a price and it is not zero. A nested Tenant, or an app of kind Kubernetes whose own hook spends two 120s waits first, misses 90s and converges across uninstall retries instead; neither fits the 300s budget as it stands either. Between those and the fast case sits a band that does pay: a wave inherits its slowest child, so one that settles around 95s completes inside the budget today and expires here instead, trading a clean attempt for one more retry. An unbounded wave does not fail, it spends the whole budget and reports a timeout naming nothing, so the trade is worth taking. Give the Job an activeDeadlineSeconds of 270. It bounds what the waits do not — the DELETE calls carry kubectl's default --request-timeout of 0 — and sits under the 300s uninstall budget flux applies when the HelmRelease carries no timeout of its own, because the deadline runs from the Job's startTime while that budget runs from just before the Job is created. At equal values helm gives up first, on a Job still running. Cap the pod's termination at 10s, which is part of that arithmetic rather than a detail of it. Since 1.31 the Job controller withholds the terminal condition until every pod has finished terminating, so helm learns of the failure at the deadline plus this period; the 30s default would land it at 270+30, level with the budget the deadline is set under. The period is spent in full: the SIGTERM goes to PID 1, which is a shell running a multi-command script, and a default disposition sent to PID 1 is discarded by the kernel with no handler installed to catch it. kubectl is a child and is never signalled. Set backoffLimit to 0 and restartPolicy to Never. The limit is needed under either policy, since the default of 6 keeps the failure from reaching helm inside the budget. The policy matters because under OnFailure the pod survives its own failure and the kubelet re-runs the container in place, with the controller applying the cap only afterwards by summing restartCount: the script re-enters the wave that just expired and is cut down mid-run. Never leaves a failed pod the controller sees directly. Add resource requests to match the sibling hook in packages/apps/kubernetes. Without them the pod is BestEffort and first out under node pressure, which a retry inside the Job used to absorb and a single attempt does not. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
Cozystack
Cozystack is a free platform and framework for building clouds.
Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.