Ivan Okhotnikov
287dbca600
kubernetes-previous.bats intermittently fails with `zfs create -V ... returned with exitcode 1: cannot create '...': out of space` on a satellite even though the LINSTOR pool measured 30+ minutes later shows 100+ GiB free per satellite. Root cause proven from a satellite ErrorReport captured in the cozyreport artefact (PR #3044 run 28751310913, report id 6A4AADFD-349B2-000000): the previous tenant Kubernetes test's cozy_wait_tenant_drained gate returned as soon as its PVCs disappeared from the API, but the satellite-side ZFS `zvol destroy` for the just-deleted replicated (autoPlace=3, DRBD) 20 GiB worker disks lagged tens of seconds behind that signal as DRBD adjusted, unref counts drained, and ZFS batch-destroyed the datasets. The next tenant test's CreateVolume then landed inside that window against a pool that had not yet reclaimed the space, LINSTOR-CSI retried autoplace against its own still-being-torn-down previous placement, and worker Machine bringup stretched past the MHC nodeStartupTimeout budget with no cluster-side fault. Add a companion `cozy_wait_linstor_pool_free` helper that polls the LINSTOR controller for the smallest FreeCapacity across every ZFS satellite pool and blocks until it reaches a per-satellite threshold (default 90 GiB — covers the ~82 GiB worst-case footprint of two 20 GiB replicated worker targets across all three satellites plus two 21 GiB CDI scratch PVCs landing on the same node via the local storageClass). Called from cozy_cleanup immediately after cozy_wait_tenant_drained. Bounded and best-effort: 300 s timeout, `|| true` at the call site, warning printed with the observed pool state if the threshold is not reached in budget so the next test does not misdiagnose the pressure. The threshold is a floor for the concrete failure mode observed in the E2E suite; the surrounding kubernetes-latest -> kubernetes-previous handoff is the only run in the current suite that immediately follows a heavy tenant teardown with another tenant provisioning, so this gate is the natural place for the check. E2E-scope only, no product or chart change. The proximate observation (`zfs create out of space`) surfaced because of PR #3209 which taught hack/cozyreport.sh to bundle satellite ErrorReports; without that artefact the root cause was ambiguous between multiple plausible LINSTOR internal races. Ships alongside that PR. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Cozystack
Cozystack is a free platform and framework for building clouds.
Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.