mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-03 18:47:04 +09:00
fix(e2e): widen tenant-root monitoring bring-up waits to 15m under load (#3160)
## What this PR does The `Configure Tenant and wait for applications` install test brings up the entire root-tenant monitoring stack and waits for each component to become ready/operational. Those waits span the VictoriaMetrics operator resources (`vmalert`/`vmalertmanager`, `vlclusters/generic`, `vmcluster/shortterm`+`longterm`) and the Grafana pair (`grafana-db` CNPG cluster, `grafana-deployment`). Each converges only once its underlying workloads schedule and turn Ready, and during platform bring-up they contend for the same node resources as the rest of the install — so their convergence is load-sensitive. Only `vmalert`/`vmalertmanager` used a 15m budget; the other four used 10m, so this block carried a 10m/15m split even though all its resources contend for the same node capacity during install. Under install-time load (multiple concurrent e2e sandboxes on one runner) the monitoring bring-up is slow across the whole block rather than on a single resource, so the shorter budgets risk failing the whole install on PRs whose branches never touched monitoring — and per-resource bumps are whack-a-mole: raising one budget just moves the risk to the next component in the block. On a calm sandbox each of these reaches ready/operational in under a second, which is what distinguishes the two cases — slow-not-stuck convergence under contention, not a product regression. This lifts the whole tenant-root monitoring wait-set to a single uniform 15m budget and wraps each wait in an on-timeout diagnostic block that dumps the resource YAML (status conditions / `updateStatus`) plus tenant-root pod state. A genuinely stuck resource still fails — now at 15m, and with legible diagnostics instead of a bare `timed out` line — so this does not weaken the gate's ability to catch real breakage. The extra budget costs near-zero wall-time on the happy path and stays well inside the E2E job's 180m cap. The remaining upstream angle is the real driver: how long the VictoriaMetrics/VictoriaLogs stack and the Grafana bring-up take to become operational under resource contention during install. This change addresses the e2e robustness side; reducing that bring-up cost is a separate, product-side improvement. ### Screenshots Not applicable — no UI changes. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved and standardized readiness checks during tenant setup so application bring-up waits behave more consistently. * Enhanced failure handling to provide clearer diagnostics (including resource details and pod status) when readiness checks time out. * **Tests** * Standardized longer readiness wait timeouts for multiple application components to reduce flakiness during e2e runs. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
1 file changed
+57
-7
@@ -213,20 +213,70 @@ EOF
|
||||
timeout 60 sh -ec 'until kubectl get sts/etcd -n tenant-root >/dev/null 2>&1; do sleep 2; done'
|
||||
kubectl wait sts/etcd -n tenant-root --for=jsonpath='{.status.readyReplicas}'=3 --timeout=10m
|
||||
|
||||
# VictoriaMetrics components
|
||||
# VictoriaMetrics components. vmalert/vmalertmanager, vlclusters/generic and
|
||||
# vmcluster/shortterm+longterm are all vm-operator-managed resources that flip
|
||||
# updateStatus=operational only once their workloads are scheduled and Ready.
|
||||
# During platform bring-up they contend for node resources with the rest of
|
||||
# the install, so convergence is load-sensitive: on a calm sandbox each reaches
|
||||
# operational in under a second, but under install-time load (concurrent e2e
|
||||
# sandboxes on one runner) monitoring bring-up is slow. vmalert already uses a
|
||||
# 15m budget; vlclusters and vmcluster used 10m, so this block carried a
|
||||
# 10m/15m split even though all three contend for the same node capacity and a
|
||||
# slow VictoriaLogs bring-up can fail the install on a PR that never touched
|
||||
# monitoring. Unify the block on one 15m budget (near-zero cost in the happy
|
||||
# path, comfortably inside the E2E job budget) and dump live status on timeout
|
||||
# so a genuine stuck-not-slow regression stays legible instead of surfacing as
|
||||
# a bare "timed out" line.
|
||||
timeout 60 sh -ec 'until kubectl get vmalert/vmalert-shortterm -n tenant-root >/dev/null 2>&1; do sleep 2; done'
|
||||
timeout 60 sh -ec 'until kubectl get vmalertmanager/alertmanager -n tenant-root >/dev/null 2>&1; do sleep 2; done'
|
||||
kubectl wait vmalert/vmalert-shortterm vmalertmanager/alertmanager -n tenant-root --for=jsonpath='{.status.updateStatus}'=operational --timeout=15m
|
||||
kubectl wait vmalert/vmalert-shortterm vmalertmanager/alertmanager -n tenant-root --for=jsonpath='{.status.updateStatus}'=operational --timeout=15m || {
|
||||
echo "=== vmalert/vmalert-shortterm, vmalertmanager/alertmanager did not reach updateStatus=operational ==="
|
||||
kubectl get vmalert/vmalert-shortterm vmalertmanager/alertmanager -n tenant-root -o yaml 2>&1 || true
|
||||
echo "=== tenant-root pods ==="
|
||||
kubectl get pods -n tenant-root -o wide 2>&1 || true
|
||||
false
|
||||
}
|
||||
timeout 60 sh -ec 'until kubectl get vlclusters/generic -n tenant-root >/dev/null 2>&1; do sleep 2; done'
|
||||
kubectl wait vlclusters/generic -n tenant-root --for=jsonpath='{.status.updateStatus}'=operational --timeout=10m
|
||||
kubectl wait vlclusters/generic -n tenant-root --for=jsonpath='{.status.updateStatus}'=operational --timeout=15m || {
|
||||
echo "=== vlclusters/generic did not reach updateStatus=operational ==="
|
||||
kubectl get vlclusters/generic -n tenant-root -o yaml 2>&1 || true
|
||||
echo "=== tenant-root pods ==="
|
||||
kubectl get pods -n tenant-root -o wide 2>&1 || true
|
||||
false
|
||||
}
|
||||
timeout 60 sh -ec 'until kubectl get vmcluster/shortterm vmcluster/longterm -n tenant-root >/dev/null 2>&1; do sleep 2; done'
|
||||
kubectl wait vmcluster/shortterm vmcluster/longterm -n tenant-root --for=jsonpath='{.status.updateStatus}'=operational --timeout=10m
|
||||
kubectl wait vmcluster/shortterm vmcluster/longterm -n tenant-root --for=jsonpath='{.status.updateStatus}'=operational --timeout=15m || {
|
||||
echo "=== vmcluster/shortterm,longterm did not reach updateStatus=operational ==="
|
||||
kubectl get vmcluster/shortterm vmcluster/longterm -n tenant-root -o yaml 2>&1 || true
|
||||
echo "=== tenant-root pods ==="
|
||||
kubectl get pods -n tenant-root -o wide 2>&1 || true
|
||||
false
|
||||
}
|
||||
|
||||
# Grafana
|
||||
# Grafana. The grafana-db CNPG cluster and the grafana-deployment Deployment
|
||||
# complete the tenant-root monitoring bring-up and contend for the same node
|
||||
# resources as the VictoriaMetrics stack above during install. Under
|
||||
# install-time load (concurrent e2e sandboxes on one runner) either can be slow
|
||||
# and fail the install on a PR that never touched monitoring, so both move from
|
||||
# their 10m budget to the same uniform 15m as the vm-operator waits above and
|
||||
# dump live status on timeout to keep a genuine stuck-not-slow regression
|
||||
# legible instead of surfacing as a bare "timed out" line.
|
||||
timeout 60 sh -ec 'until kubectl get clusters.postgresql.cnpg.io/grafana-db -n tenant-root >/dev/null 2>&1; do sleep 2; done'
|
||||
kubectl wait clusters.postgresql.cnpg.io/grafana-db -n tenant-root --for=condition=ready --timeout=10m
|
||||
kubectl wait clusters.postgresql.cnpg.io/grafana-db -n tenant-root --for=condition=ready --timeout=15m || {
|
||||
echo "=== clusters.postgresql.cnpg.io/grafana-db did not reach condition=ready ==="
|
||||
kubectl get clusters.postgresql.cnpg.io/grafana-db -n tenant-root -o yaml 2>&1 || true
|
||||
echo "=== tenant-root pods ==="
|
||||
kubectl get pods -n tenant-root -o wide 2>&1 || true
|
||||
false
|
||||
}
|
||||
timeout 60 sh -ec 'until kubectl get deploy/grafana-deployment -n tenant-root >/dev/null 2>&1; do sleep 2; done'
|
||||
kubectl wait deploy/grafana-deployment -n tenant-root --for=condition=available --timeout=10m
|
||||
kubectl wait deploy/grafana-deployment -n tenant-root --for=condition=available --timeout=15m || {
|
||||
echo "=== deploy/grafana-deployment did not reach condition=available ==="
|
||||
kubectl get deploy/grafana-deployment -n tenant-root -o yaml 2>&1 || true
|
||||
echo "=== tenant-root pods ==="
|
||||
kubectl get pods -n tenant-root -o wide 2>&1 || true
|
||||
false
|
||||
}
|
||||
|
||||
# Verify Grafana via ingress
|
||||
ingress_ip=$(kubectl get svc root-ingress-controller -n tenant-root -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
|
||||
|
||||
Reference in new issue
Block a user