The post-install prep script waits out a serial reconcile chain --
the linstor HelmRelease, then the linstor-controller Deployment --
against a single 15-minute deadline anchored at the script's start.
The head of the chain therefore eats the window and the tail inherits
whatever remains, so an install fails whenever the chain merely runs
long rather than never converging.
Measured on one such failure: the HelmRelease went Ready 452s in, and
the Deployment needed a further ~520s because the controller's DB
migration lost its apiserver connection and retried four times before
succeeding. Neither figure exceeds 900s on its own; only their sum
did, and the install reported the Deployment as though it were broken.
Give each wait its own budget, started when the link before it
completed -- the shape the LINSTOR node and MetalLB CRD waits further
down already have. What that buys is not speed but a guarantee: every
link gets its whole allowance regardless of what the links ahead of
it spent, so the tail can no longer be failed by the head being slow.
It is a trade, and the other half is worth stating. The script's worst
case becomes 900 + 900 + 300 + 300 = 40m against the 180m the job
carries, where before the two waits shared one 15m window. And a link
that is genuinely stuck is reported later rather than sooner: a
Deployment that never converges surfaces at 1355s where the shared
window surfaced it at 900s, on that run's timings. A HelmRelease that
never becomes Ready surfaces at the same moment as before, since its
budget still starts with the script.
A timeout now also lists the linstor pods and the namespace events.
The description names the object waited on, not the link that held it
up, and the same "Deployment not Available" has been reached both with
the controller pod crash-looping its migration init container and with
that pod blocked on a client TLS Secret cert-manager had not issued
yet. Telling those apart previously required the run's bundle.
The events come from `kubectl events` rather than `kubectl get
events`. It orders by when an event was last seen, so a condition
repeating for minutes -- what a stuck mount looks like -- stays inside
the window instead of being pushed out by newer one-off events, and it
carries no --sort-by, so it cannot fail the whole read the way a sort
key does when one item lacks it.
Both reads keep their stderr and carry a bound. A read that fails
silently is indistinguishable from a namespace with nothing to report,
which is the distinction being drawn; and the caller is blocked in
`wait`, so an unbounded read against a wedged apiserver would hold the
install open until the job's own timeout. The client budget is
strictly the smaller of the two so kubectl gets to name the reason
before the outer kill takes it away.
The wait helper is named for linstor because those diagnostics read
one namespace unconditionally: the wait is generic over its kubectl
arguments, what it dumps on the way out is not.
The new unit test runs the script end to end against a stub PATH
carrying a virtual clock, so a 15-minute budget elapses in a few
hundred forks and the timings are exact rather than wall-clock
approximate.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The post-install-prep node-list loop dialed the linstor-controller Service
(linstor+ssl://linstor-controller:3371) as soon as the controller Deployment
reported Available once. That gate goes stale: the controller carries
reloader.stakater.com/auto=true and mounts the cert-manager-issued API-TLS
Secret, so when cert-manager (re)writes that Secret during bring-up, reloader
rolls the single-replica controller. The roll drops the Service to zero ready
endpoints after Available already passed, and a dial into that window fails
with "[Errno 113] No route to host" (EHOSTUNREACH). The blind until-loop then
churned the CLI on that error instead of waiting for a routable controller.
Gate every probe on the linstor-controller Service having a ready endpoint, so
the CLI only dials a reachable controller and a mid-loop reload re-waits for
the endpoint instead of erroring. The "Online == 3" satellite-convergence
assertion and the dedicated 300s budget are preserved; a timeout now dumps
controller endpoints and pods for diagnostics before failing.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback on hack/e2e-post-install-prep.sh:31:
POSIX `wait` without arguments returns 0 even if a backgrounded
`kubectl exec` failed, which would mask partial LINSTOR prep failures
and let downstream tests run on a half-provisioned cluster. Collect
each background PID and `wait "$pid"` per job so `set -e` exits the
script on the first failed pool creation.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Two complementary changes that depend on the operator's
`--helmrelease-interval` flag landing via #2509:
1. Installer chart exposes `cozystackOperator.helmReleaseInterval`
(default empty -> flag not rendered -> operator default 5m
applies). E2E install passes `30s`. Eliminates the 8-10 min Flux
dead zone where dependency-blocked HRs (waiting on cert-manager
webhooks, CRDs) are requeued only every 5 min.
2. LINSTOR / StorageClass / MetalLB setup, previously sequential
`@test` cases after the 15 m platform HR wait, now run as a
background prep that completes during the platform reconcile.
None of them gates platform HR reconcile, so they can move out
of the critical path. Helper script `hack/e2e-post-install-prep.sh`
polls for its prerequisites and is awaited at the end of the
`Install Cozystack` test so failures still surface deterministically.
Includes the T2-race fix: wait on the LINSTOR HR being `Ready`
before polling for `deploy/linstor-controller`. The old 5-min hard
timeout fired during cold installs where the LINSTOR HR's dependency
chain takes longer than 5 min to resolve.
Production defaults are unchanged. The 30 s interval is opt-in per
install command.
Depends on:
- #2509 (operator-side `--helmrelease-interval` flag).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>