Flux helm-controller v1.5.0 (shipped in Flux v2.8) replaced the legacy
Helm readiness logic with kstatus-based health checking, which polls
Deployment/StatefulSet/Service status and waits for kstatus to assess
rollout-ready. This typically adds 10-30s of latency between resource
apply and HR condition Ready=True compared to v2.7.
Our per-app bats tests had `kubectl wait hr/... --for=condition=ready
--timeout=` budgets of 20s, 30s, 60s, 100s — sized for v2.7's faster
readiness flip and previously masked by the 3x retry on Run E2E tests
(dropped in #2558). Under v2.8 with no retry, every app that runs after
the first few hit its HR wait timeout exactly, surfacing as 13 of 16
test failures on PR #2619.
Standardise on 5m for HR Ready across all app tests. Tight enough to
catch genuine bring-up failures within the per-app job budget, loose
enough to absorb kstatus dispatch jitter on a management cluster that
is concurrently reconciling multiple tenant-test HRs.
Post-HR-Ready checks for app-specific resources (StatefulSet/Service/
PVC/operator CRs) remain — for CR-based apps kstatus only observes
that the chart applied, not that the operator finished. Audit of
which of those are actually redundant under v2.8 kstatus is queued
as a follow-up.
Also bumps the 10s "kamajicontrolplane appears" wait in
run-kubernetes.sh to 2m, and the kubernetes-addon HR Ready loop from
1m to 5m for the same reason.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 5ef02ee7b3)
Sweep across `hack/*.bats` and `hack/*.sh` replacing fragile
`sleep N` / `timeout N until …; do sleep …; done` patterns with
`kubectl wait --for=condition=…` plus a small existence backstop:
until kubectl get <resource> >/dev/null 2>&1; do sleep 2; done
kubectl wait <resource> --for=condition=Ready --timeout=…
Existence backstop added to 33 `kubectl wait` sites across 12 bats
files (redis, kafka, openbao, harbor, mongodb, qdrant, vminstance,
clickhouse, mariadb, postgres, bucket, install). Without it,
`kubectl wait` against a not-yet-created resource sits silently for
the full timeout; with it, the test fails fast with a clear
"resource never appeared" error.
OpenAPI tests' fixed `sleep 5` after starting `kubectl proxy` is now
a port-readiness wait. The 3 remaining genuine sleeps (LINSTOR node
membership propagation, MetalLB end-to-end HTTP path, HR-count
emission heuristic) are annotated with TODO + rationale so a future
reader knows they are intentional.
Same green-path semantics; failures now surface in seconds instead
of after a 5-minute timeout.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
- Add pre-cleanup of stale resources to all app E2E tests so
retries start fresh instead of patching stuck state
- Wait for port-forward to be ready before using it in
kubernetes tests (fixes race condition)
- Reduce version check sleep from 5s to 1s for faster retries
- Clean up temp directory on test failure in cozytest.sh
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Myasnikov Daniil