Aleksei Sviridkin
ab8675161c
## What this PR does Three of the six steps in the `kubernetes` pre-delete hook call `kubectl delete` with neither `--wait=false` nor `--timeout`. `--wait` defaults to true and `--timeout=0` is implemented as a week, so the KamajiControlPlane delete in step 3, the DataVolume delete in step 5 and the LoadBalancer Service delete in step 6 each park the hook for 168 hours on one stuck finalizer. A DataVolume held by a CDI importer or a PVC finalizer is the reachable case. The waits in steps 2 and 4 were bounded earlier, in response to a report that named the child-HelmRelease wait and nothing else. The fix stopped where the report did, so the shape it described stayed in the three steps nobody had looked at. None of the three waits now. Nothing downstream reads the result of steps 5 and 6. What matters after step 3 is the TenantControlPlane, and step 4 deletes that one directly and waits for it, so blocking on the KamajiControlPlane first only adds a second place for a finalizer to stall. An expired wait and `--wait=false` leave the same cluster state, with the same objects finalizing in the background. Not waiting also keeps these steps under `set -e` with no guard, which matters more than it looks. A guard cannot tell an expired wait from a rejected request, so it would report a failed delete as cleanup that happened. With no timeout to expire, every non-zero exit left is a real failure. The `watch` verb goes away for those three resources. kubectl opens a watch only on the wait path, so the grant was unexercised. `helmreleases` and `tenantcontrolplanes` keep it because steps 2 and 4 still wait, and there the verb is load-bearing in a way that is easy to miss: removing it does not produce an error, it hangs the delete. That is the failure this repo already hit once at the tenant level, so the whole rule set is pinned rather than left to the next reviewer of this Role. `backoffLimit` goes to 0, against a default of 6 that `restartPolicy: Never` turns into seven pods in series, each restarting the script from step 1 and each paying its own image pull. ## What this PR does not do It does not add `activeDeadlineSeconds`, and that is arithmetic rather than oversight. A deadline has to sit above the sum of the remaining waits, or it cuts runs that were going to finish, and far enough under helm-controller's uninstall budget to be the thing helm reports rather than something helm waits through. Those two bounds do not currently leave a gap, and the image pull comes from Docker Hub with no mirror in front of it. Room comes from raising `spec.uninstall.timeout` on the generated HelmRelease first; squeezing the waits further is the wrong direction. Worth stating plainly, because it is the one change here that is not purely a tightening: with `backoffLimit: 0` a pod eviction now fails the whole uninstall attempt, where six retries previously absorbed it. That is deliberate. Both remaining waits are guarded, so an attempt can spend its entire budget without failing, and a retry would restart from step 1 with the clock already there; on a nodeless tenant it would also re-pay step 4, since the first attempt leaves the TenantControlPlane still terminating. The failure a retry would cover is an eviction, which is not bounded in time and costs most when it arrives late. Flux still repeats the uninstall on its own interval, so the end state is preserved and the trade is wall-clock for predictability. It does not bound the requests. `--wait=false` and `--timeout` bound the waiting, so an apiserver that accepts the connection and then stops answering hangs this hook exactly as before. That is a different failure from the stuck finalizer, it is not made worse here, and it wants one answer across every call rather than a flag on the deletes. ## Tests `packages/apps/kubernetes/tests/delete_hook_test.yaml` gains cases for the bounds and for the Role. Every delete in the hook is asserted to carry one, the Job is pinned to a single attempt, and the rule set is pinned whole. The regex patterns walk only continuation lines, the ones ending in a backslash, so each stays inside the command it names. A cross-line wildcard reaches a flag belonging to the next command instead, and RE2 has no lookahead to exclude the `echo` in between. The Role assertions use whole-rule `contains` plus `lengthEqual` rather than per-index verb checks. `notContains` passes when its path does not resolve, so an index-addressed "and nowhere else" goes quiet the moment the rules list is reordered or shortened, which is exactly the edit it exists to catch. Matching the entire rule pins the verb set in both directions and names the resource when it fails. Every pinned value was mutation-tested: dropping `--wait=false` from each of the three deletes, dropping `--timeout` from either of the two that still wait, `backoffLimit` in both directions, adding `watch` back, removing `watch` from a resource that waits, appending a rule past everything asserted, and reordering the rules list. Each one fails the assertion that claims that ground and no other. ### Screenshots None. This changes a Job manifest and its tests, with no user-visible surface. ### Downstream repositories Walked the trigger map against the diff. The diff is one Job manifest inside the kubernetes chart and its helm-unittest suite. No package is added, renamed or removed; no `values.schema.json`, `values.yaml` default or version enum changes; no `ApplicationDefinition` semantics, namespace, variant or release asset changes; nothing under `hack/` is touched and no make target changes behaviour. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kubernetes): bound the pre-delete hook's remaining unbounded deletions and limit its Job to a single attempt, so a stuck finalizer on a DataVolume, LoadBalancer Service or KamajiControlPlane no longer parks the hook and blocks the uninstall ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved Kubernetes resource cleanup during deletion to avoid indefinite waits. * Cleanup operations now complete with bounded waiting and fewer retry attempts, improving reliability and responsiveness. * Removed unnecessary access permissions from cleanup operations, strengthening security. * **Tests** * Added coverage to verify deletion timeouts, retry behavior, and required permissions across supported resources. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Cozystack
Cozystack is a free platform and framework for building clouds.
Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.