Aleksei Sviridkin
89c7ea0b46
## What this PR does This fixes two cleanup failures in the kubernetes suites. The nightly runs failed `kubernetes-previous` and `kubernetes-latest` in the replicated storage lane. PR runs fail `kubernetes-latest` with `LINSTOR pools did not return to their pre-suite baseline within 300s`. The replicated lane asked for too much free space at cleanup. It wanted 90 GiB free on every LINSTOR node, and volumes from earlier suites kept srv2 at 83 GiB, so both suites failed without leaking anything (nightly run 36659663201). A replicated suite still needs those 90 GiB for its workers and CDI scratch volumes, so the floor is now checked before the suite starts. If a node is short there, the suite fails with the per-node table and an error that blames the sandbox, not the suite. Cleanup in both lanes now compares against a baseline recorded before the suite creates anything, and that baseline is no longer free bytes. On a node whose pools are all thick ZFS, `linstor sp l` does not ask the satellite for free space. The controller fetches it live only for nodes with thin pools and otherwise returns the value cached at the satellite's last report. In LINSTOR v1.33.3 that is `FreeCapacityFetcherProto.assembleRequests` filtering on `hasThinPools`, with `CtrlStorPoolListApiCallHandler` falling back to `getFreeCapacityLastUpdated`. The last report can be taken while `zfs destroy` is still freeing blocks in the background. A cleanup that had removed everything then kept polling a frozen figure 0.4 to 0.9 GiB short and failed after 300s. A byte tolerance does not fix that, because the residue is already close to the smallest volume a suite is known to have leaked (1 GiB). The baseline is now the set of ZFS datasets under each node's LINSTOR pool. It is read with `zfs list` inside that node's `linstor-satellite` container. The zpool name comes from the pool's `StorDriver/StorPoolName` property, because it is `data` on QEMU nodes and `data-srvN` in container mode. Cleanup waits until no dataset outside the baseline remains. A destroyed dataset leaves the namespace when `zfs destroy` returns, whether or not its blocks are still being freed, so background freeing no longer fails the check. A real leak, like the 21 GiB scratch volume left on srv3 in an earlier run, is reported by node and dataset name instead of a KiB difference. A node whose pool cannot be read fails the check. A run that fails before it records a baseline no longer compares against the previous suite's file. Every read the new probe makes runs under `timeout -k 5 30`, so a hung controller or satellite fails one poll and the wait still ends with its failure message. The kafka suite deleted its apps but never their PVCs. The chart keeps them on purpose with `deleteClaim: false`. One run left 35 GiB in `tenant-test`, 20 GiB of it on srv2, and the backup round trip left its two clusters' volumes in `tenant-root` the same way. Each test now deletes its clusters' PVCs by `strimzi.io/cluster` after the apps are gone and waits for them. The `kafka` test does it from a step cleanup, so it also runs on a failed run. The tests cover both fixes. `hack/linstor-storage-mode_test.bats` has cases for a clean cleanup, a volume still listed on the first poll and gone on the next while the cached free capacity stays short, a leaked volume, a node that cannot be read, a hung satellite read, a missing or stale baseline, and the replicated-lane floor. Its fake runs the real controller-side `jq` filter against a LINSTOR-shaped listing. `hack/kafka-suite-pvc-cleanup_test.bats` checks that each cluster's PVC delete is in the suite and runs after the app cleanup. One stub line goes into `hack/run-kubernetes-talos-diagnostics_test.bats`, because that test calls `cozy_cleanup` with no baseline and only checks the order of the deletes. The new guards were mutation-tested. Some limits stay as they are. The local lane waits for dataset names to disappear, not for the freed space, and has no free-space floor, so a background destroy can still be freeing blocks when the next suite starts. The 90 GiB precondition in the replicated lane still reads the cached free capacity, and its comment says so. The two `kubectl exec` reads in `cozy_wait_linstor_pool_free` were there before this PR and still have no time limit of their own. Two related problems stay out of this PR. The `kafka-metadata` suite leaks PVCs the same way, tracked in #4641. The stale comment in `examples/backups/kafka/cleanup.sh` and the missing PVC pre-clean in the round trip are in #4649. Closes #4617, closes #4616, closes #4653 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(e2e): the kubernetes suites' LINSTOR cleanup compares each node's ZFS datasets against a pre-suite baseline instead of cached free capacity, the replicated lane checks its 90 GiB floor before the suite starts, and the kafka suite deletes the PVCs its clusters keep. ```
Cozystack
Cozystack is a free and open-source platform and framework for building clouds.
Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.
Project documents
| Document | What it covers |
|---|---|
| Governance | How decisions are made and who makes them |
| Contributor Ladder | How to grow from contributor to maintainer |
| Maintainers | Who maintains the project |
| Code of Conduct | What we expect of each other |
| Security Policy | Reporting a vulnerability, and which releases get fixes |
| Adopters | Who runs Cozystack in production |
| Roadmap | Where the project is going |
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.