mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-03 02:27:01 +09:00
Aleksei Sviridkin
479929ebfb
## What this PR does
Drops the 3× retry loop on `Run E2E tests` and `Install Cozystack into
sandbox`. `Prepare environment` keeps its 3× retry — that step is pure
infrastructure (Talos image download, sandbox VM boot, network) where
transient runner hiccups warrant a retry.
On failure, the test step now captures `kubectl get hr -A -o wide` and
`kubectl get events -A` under a collapsible group so triage starts with
the actual broken-state snapshot.
> [!NOTE]
> An earlier revision of this PR also doubled every bats timeout. That
commit was dropped in a rebase and is intentionally **not restored**:
the timeout class that actually matters (per-app HR-Ready waits) has
since been standardized at 5m on `main` (7b9f286fa), making a blanket 2×
redundant.
**Fixes gateway.bats teardown leakage.** The nested-tenant tests deleted
tenants fire-and-forget, parent and child back-to-back. The leftover
uninstalls (each blocked on a cleanup Job, parents wedged on
still-terminating child namespaces) plus one mid-install child HR
occupied exactly 5 workers on the `--concurrent=5` tenants
helm-controller shard, starving whichever app test ran next — observed
as the harbor HR sitting unreconciled for its whole 5m HR-Ready budget
in [run
27020081550](https://github.com/cozystack/cozystack/actions/runs/27020081550),
surfaced by this PR's own retry removal + diagnostics dump. Teardown now
deletes child→parent with hard `wait hr --for=delete` between, so a
wedged tenant uninstall fails gateway.bats itself, not an innocent
neighbor.
## Why
Audit of 30 successful PR runs found that across 5 sampled failure
attempts, **25/25 retries** on `Run E2E tests` failed — the retry loop
never recovered a flake, only stretched deterministic failures and
tripled diagnostic wall-time. Same data shape on `Install Cozystack`.
Beyond wasted CI time, the retry was hiding ~10 deterministic bugs (Helm
namespace-ownership conflict, seaweedfs HR timeout, harbor BucketInfo
wiring, vminstance disk race, etc.). Each failure looked like a "flake"
because the retry sometimes coincided with whatever transient state had
cleared — the retry never fixed the bug, just delayed surfacing.
## Dependencies
The deterministic bugs the retry was masking are now fixed on `main`:
- ✅ **#2508** — installer namespace bootstrap (Helm namespace-ownership
conflict on cold install) — merged
- ✅ **#2509** — operator HelmRelease config knobs (`seaweedfs-system`
2-min wait race within Flux's 5-min reconcile windows) — merged
- ✅ **#2528** — harbor bucket-secret + BucketInfo gating (harbor
`ValuesError` on first install) — merged
- ✅ **#2529** — objectstorage-controller BucketAccess conflict retry —
merged
Companion PRs in the #2619 split (independent of this PR,
ordering-wise):
- **#2602** — Flux v2.8.0 + chart fixes
- **#2601** — seaweedfs-system split
This PR does NOT depend on #2602/#2601 — it now touches only the
workflow file and gateway.bats teardown, both on top of fresh `main`.
Surfaced from #2500.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* CI prepare-environment step now reports plain attempt counts with
clear success/failure messages.
* Install and per-app test steps no longer retry; each runs once and
fails immediately on error. Failed apps log diagnostics and job proceeds
to remaining apps while overall job fails.
* **Tests**
* End-to-end tests and install/prepare flows use longer, more tolerant
timeouts and added existence polling to reduce flakiness and improve
diagnostics.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2558?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
