mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-04 02:57:27 +09:00
Andrei Kvapil
0f0204a6c2
## What this PR does The e2e QEMU guests are launched in `hack/e2e-prepare-cluster.bats` with three `-drive … if=virtio,format=raw` lines that carry no `cache=` option, so QEMU falls back to its `cache=writeback` default, which honors every guest fsync against the host-backed raw disk files. The management-cluster Talos etcd — which holds every operator's leader-election lease — fsyncs on each Raft commit. During the VM-heavy suites, containerd is simultaneously unpacking dozens of images and CDI is importing multi-GB VM disks onto the same host-backed raw files. The resulting fsync storm serializes etcd's writes into multi-second stalls, so the management apiserver returns `context deadline exceeded` on lease PUTs, and several operators (capk, etcd-operator, fdb-operator, cilium-operator, kamaji) lose leadership at the same moment. Whichever chainsaw suite is mid-flight then fails — a different one each run (vmcluster, kubernetes-latest, vmdisk), which is the fingerprint of shared-resource contention rather than a bug in any single suite. This adds `cache=unsafe` to each guest drive. That makes guest flushes no-ops against the host page cache, which removes the fsync-gated etcd stalls at the root of the flakiness. It is safe here precisely because these CI VMs are ephemeral and destroyed at the end of every run — a crash before writeback loses only throwaway state — and it must never be used for real data. The change is minimal and uniform across all three guest drives; no non-guest drive is touched (there are none in this file). Runner sizing alone (#3251) did not resolve this, because more CPU and memory do not change the I/O path: the guest fsyncs still propagate through `cache=writeback` to the host, and etcd still stalls behind the same fsync storm. This addresses the I/O path directly. `git log -S 'cache='` on this file is empty, so a cache mode has never previously been set here. ### Screenshots Not applicable — no UI changes. ### Downstream repositories I walked the trigger map in `docs/agents/contributing.md` against the actual diff. The only entries that reference `hack/e2e-prepare-cluster.bats` (ansible-cozystack and ccp) are scoped to node prerequisites — kernel modules, containerd settings, the LVM `global_filter`, and the cluster domain. This change touches none of those: it adds a QEMU host-side hypervisor flag (`cache=unsafe`) to the CI VM `-drive` options, which is invisible to the guest OS and to any real node. No downstream repository is affected. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(e2e): boot e2e QEMU guests with cache=unsafe so ephemeral CI VM fsyncs no longer stall the management-cluster etcd, removing the cross-operator leader-election flakiness that failed a different chainsaw suite each run ``` --- This PR was prepared with the assistance of an AI agent (Claude). A human maintainer reviewed the diff and the reasoning before submission and remains responsible for the change. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Updated end-to-end virtual machine startup testing to use optimized QEMU disk caching settings for improved reliability and performance. * Improved end-to-end Kubernetes cluster readiness checks under load by extending the deadline for reaching “at least 2 nodes Ready,” and aligned related messaging and failure diagnostics accordingly. <!-- end of auto-generated comment: release notes by coderabbit.ai -->