The shell translator rewrites test bodies and loses Bats reporting.
The unit lane needs complete TAP results and JUnit reports without
changing how live-cluster suites run.
Sourced production helpers still need a dash pass, and unit test bodies
need nounset without exporting shell options into subprocesses.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
When the outer timeout cut a `kubectl wait` attempt, nothing was
printed: `timeout` is silent and kubectl dies on the signal. The
timeout line then read "last kubectl error: none", which says the
client had nothing to report rather than that it was killed
mid-request. The attempt's exit status now names that case.
The errors the loop prints also carry the script's prefix. The first
NotFound shows up on every green run, since the release appears about
a minute in, and without the prefix it reads like a failure when the
log is scanned after one.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
`kubectl wait --timeout` looked like a bound on the wait loop's calls,
but it covers only the watch. The command first resolves the object
with a plain GET that has no deadline, since no --request-timeout is
set, so an apiserver that accepts the connection and never answers
holds the wait loop, the most frequently repeated call in the script,
just like the calls that were unbounded outright.
Each attempt now runs under an outer timeout set a little above the
attempt's own --timeout, so the wait still owns its normal expiry.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The storage pool listing was piped straight into awk, so its exit
status was lost: a listing that failed, or one killed by its time
bound, read as "no pools exist yet". The script then sent a create for
every node, and on the nodes whose pool did exist the create failed
with a reason unrelated to the actual failure.
The listing now runs on its own, so a failure stops the script before
any pool is created. A bound that fires leaves no text of its own:
`timeout` prints nothing and kubectl dies on the signal before it can.
The bounded calls therefore go through a wrapper that names the call
it killed, so the log does not end on the step's announcement line.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Both waits ran as `timeout N sh -ec 'until kubectl get ...'` with the
read's output discarded. On elapse `timeout` exits 124 and `set -e`
ends the script, so the last line in the log is the one announcing the
wait: CRDs that are late and an operator that never got far enough to
install them read the same.
A shared deadline loop now polls quietly and, on the way out, reads the
objects once more with stderr kept, so the log names which object is
missing. The MetalLB wait also lists the metallb HelmRelease, whose
state separates those two cases. The loop runs on the script's own
clock, which lets the unit suite drive it with the virtual one.
The unit suite's runaway guard in the sleep stub now kills its caller
as well, because a failing exit status does not stop a loop that runs
where set -e is off, and a guard that cannot stop the loop turns a
broken deadline into a hung test.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
When the linstor HelmRelease never goes Ready, the script reported a
timeout on linstor and dumped the cozy-linstor pods and events. In two
failing install runs the linstor release only sat behind a dependency
chain (kubeovn -> piraeus-operator-crds -> piraeus-operator -> linstor),
so the evidence it printed described the wrong release and said
nothing about the one that was stuck.
The timeout now lists every HelmRelease first. Each Ready message names
the dependency it waits on, so the post-install-prep log names the
release at the head of the chain.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
The loop discarded stderr to stay quiet while an object is not
created yet, since `kubectl wait` exits non-zero at once on NotFound.
An expired credential or an RBAC denial exits the same way, so the loop
retried a client that could never succeed for the whole budget and
then reported a timeout on the object, pointing at the cluster while
the cause sat in the client.
Each attempt's stderr is now kept and printed whenever it changes, and
the timeout line repeats the last one. A NotFound repeated every poll
still costs a single line. The text is printed rather than matched to
fail fast: apiserver restarts during install produce the same
connection and auth errors transiently, and a match list would turn a
slow link into a false failure.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The install step blocks in `wait` on this script, so a kubectl call
that never returns does not fail the step: it holds it open until the
job's own ceiling cancels the run, which leaves different evidence
behind than a failed test. Only the waits carried a budget. The execs
that create storage pools, the applies, the CDI patch, and the reads
inside the LINSTOR node loop did not, and the loop's deadline is
checked between iterations, on the other side of a call that hangs.
The reads that explain a node-loop timeout were unbounded too, on the
one path that runs only when the cluster is already unhealthy.
Each call now runs under an outer timeout, and plain requests carry a
client timeout strictly inside it so kubectl names the reason before
the kill lands. The helpers take the bound as a prefix that stays empty
when unit tests source them, since `timeout` would bypass the kubectl
function those tests substitute.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The post-install prep script waits out a serial reconcile chain --
the linstor HelmRelease, then the linstor-controller Deployment --
against a single 15-minute deadline anchored at the script's start.
The head of the chain therefore eats the window and the tail inherits
whatever remains, so an install fails whenever the chain merely runs
long rather than never converging.
Measured on one such failure: the HelmRelease went Ready 452s in, and
the Deployment needed a further ~520s because the controller's DB
migration lost its apiserver connection and retried four times before
succeeding. Neither figure exceeds 900s on its own; only their sum
did, and the install reported the Deployment as though it were broken.
Give each wait its own budget, started when the link before it
completed -- the shape the LINSTOR node and MetalLB CRD waits further
down already have. What that buys is not speed but a guarantee: every
link gets its whole allowance regardless of what the links ahead of
it spent, so the tail can no longer be failed by the head being slow.
It is a trade, and the other half is worth stating. The script's worst
case becomes 900 + 900 + 300 + 300 = 40m against the 180m the job
carries, where before the two waits shared one 15m window. And a link
that is genuinely stuck is reported later rather than sooner: a
Deployment that never converges surfaces at 1355s where the shared
window surfaced it at 900s, on that run's timings. A HelmRelease that
never becomes Ready surfaces at the same moment as before, since its
budget still starts with the script.
A timeout now also lists the linstor pods and the namespace events.
The description names the object waited on, not the link that held it
up, and the same "Deployment not Available" has been reached both with
the controller pod crash-looping its migration init container and with
that pod blocked on a client TLS Secret cert-manager had not issued
yet. Telling those apart previously required the run's bundle.
The events come from `kubectl events` rather than `kubectl get
events`. It orders by when an event was last seen, so a condition
repeating for minutes -- what a stuck mount looks like -- stays inside
the window instead of being pushed out by newer one-off events, and it
carries no --sort-by, so it cannot fail the whole read the way a sort
key does when one item lacks it.
Both reads keep their stderr and carry a bound. A read that fails
silently is indistinguishable from a namespace with nothing to report,
which is the distinction being drawn; and the caller is blocked in
`wait`, so an unbounded read against a wedged apiserver would hold the
install open until the job's own timeout. The client budget is
strictly the smaller of the two so kubectl gets to name the reason
before the outer kill takes it away.
The wait helper is named for linstor because those diagnostics read
one namespace unconditionally: the wait is generic over its kubectl
arguments, what it dumps on the way out is not.
The new unit test runs the script end to end against a stub PATH
carrying a virtual clock, so a 15-minute budget elapses in a few
hundred forks and the timings are exact rather than wall-clock
approximate.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Myasnikov Daniil