When the outer timeout cut a `kubectl wait` attempt, nothing was
printed: `timeout` is silent and kubectl dies on the signal. The
timeout line then read "last kubectl error: none", which says the
client had nothing to report rather than that it was killed
mid-request. The attempt's exit status now names that case.
The errors the loop prints also carry the script's prefix. The first
NotFound shows up on every green run, since the release appears about
a minute in, and without the prefix it reads like a failure when the
log is scanned after one.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
`kubectl wait --timeout` looked like a bound on the wait loop's calls,
but it covers only the watch. The command first resolves the object
with a plain GET that has no deadline, since no --request-timeout is
set, so an apiserver that accepts the connection and never answers
holds the wait loop, the most frequently repeated call in the script,
just like the calls that were unbounded outright.
Each attempt now runs under an outer timeout set a little above the
attempt's own --timeout, so the wait still owns its normal expiry.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The storage pool listing was piped straight into awk, so its exit
status was lost: a listing that failed, or one killed by its time
bound, read as "no pools exist yet". The script then sent a create for
every node, and on the nodes whose pool did exist the create failed
with a reason unrelated to the actual failure.
The listing now runs on its own, so a failure stops the script before
any pool is created. A bound that fires leaves no text of its own:
`timeout` prints nothing and kubectl dies on the signal before it can.
The bounded calls therefore go through a wrapper that names the call
it killed, so the log does not end on the step's announcement line.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Both waits ran as `timeout N sh -ec 'until kubectl get ...'` with the
read's output discarded. On elapse `timeout` exits 124 and `set -e`
ends the script, so the last line in the log is the one announcing the
wait: CRDs that are late and an operator that never got far enough to
install them read the same.
A shared deadline loop now polls quietly and, on the way out, reads the
objects once more with stderr kept, so the log names which object is
missing. The MetalLB wait also lists the metallb HelmRelease, whose
state separates those two cases. The loop runs on the script's own
clock, which lets the unit suite drive it with the virtual one.
The unit suite's runaway guard in the sleep stub now kills its caller
as well, because a failing exit status does not stop a loop that runs
where set -e is off, and a guard that cannot stop the loop turns a
broken deadline into a hung test.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
When the linstor HelmRelease never goes Ready, the script reported a
timeout on linstor and dumped the cozy-linstor pods and events. In two
failing install runs the linstor release only sat behind a dependency
chain (kubeovn -> piraeus-operator-crds -> piraeus-operator -> linstor),
so the evidence it printed described the wrong release and said
nothing about the one that was stuck.
The timeout now lists every HelmRelease first. Each Ready message names
the dependency it waits on, so the post-install-prep log names the
release at the head of the chain.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
The loop discarded stderr to stay quiet while an object is not
created yet, since `kubectl wait` exits non-zero at once on NotFound.
An expired credential or an RBAC denial exits the same way, so the loop
retried a client that could never succeed for the whole budget and
then reported a timeout on the object, pointing at the cluster while
the cause sat in the client.
Each attempt's stderr is now kept and printed whenever it changes, and
the timeout line repeats the last one. A NotFound repeated every poll
still costs a single line. The text is printed rather than matched to
fail fast: apiserver restarts during install produce the same
connection and auth errors transiently, and a match list would turn a
slow link into a false failure.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The install step blocks in `wait` on this script, so a kubectl call
that never returns does not fail the step: it holds it open until the
job's own ceiling cancels the run, which leaves different evidence
behind than a failed test. Only the waits carried a budget. The execs
that create storage pools, the applies, the CDI patch, and the reads
inside the LINSTOR node loop did not, and the loop's deadline is
checked between iterations, on the other side of a call that hangs.
The reads that explain a node-loop timeout were unbounded too, on the
one path that runs only when the cluster is already unhealthy.
Each call now runs under an outer timeout, and plain requests carry a
client timeout strictly inside it so kubectl names the reason before
the kill lands. The helpers take the bound as a prefix that stays empty
when unit tests source them, since `timeout` would bypass the kubectl
function those tests substitute.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.
The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.
Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.
Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.
The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.
Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.
One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.
Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The post-install prep script waits out a serial reconcile chain --
the linstor HelmRelease, then the linstor-controller Deployment --
against a single 15-minute deadline anchored at the script's start.
The head of the chain therefore eats the window and the tail inherits
whatever remains, so an install fails whenever the chain merely runs
long rather than never converging.
Measured on one such failure: the HelmRelease went Ready 452s in, and
the Deployment needed a further ~520s because the controller's DB
migration lost its apiserver connection and retried four times before
succeeding. Neither figure exceeds 900s on its own; only their sum
did, and the install reported the Deployment as though it were broken.
Give each wait its own budget, started when the link before it
completed -- the shape the LINSTOR node and MetalLB CRD waits further
down already have. What that buys is not speed but a guarantee: every
link gets its whole allowance regardless of what the links ahead of
it spent, so the tail can no longer be failed by the head being slow.
It is a trade, and the other half is worth stating. The script's worst
case becomes 900 + 900 + 300 + 300 = 40m against the 180m the job
carries, where before the two waits shared one 15m window. And a link
that is genuinely stuck is reported later rather than sooner: a
Deployment that never converges surfaces at 1355s where the shared
window surfaced it at 900s, on that run's timings. A HelmRelease that
never becomes Ready surfaces at the same moment as before, since its
budget still starts with the script.
A timeout now also lists the linstor pods and the namespace events.
The description names the object waited on, not the link that held it
up, and the same "Deployment not Available" has been reached both with
the controller pod crash-looping its migration init container and with
that pod blocked on a client TLS Secret cert-manager had not issued
yet. Telling those apart previously required the run's bundle.
The events come from `kubectl events` rather than `kubectl get
events`. It orders by when an event was last seen, so a condition
repeating for minutes -- what a stuck mount looks like -- stays inside
the window instead of being pushed out by newer one-off events, and it
carries no --sort-by, so it cannot fail the whole read the way a sort
key does when one item lacks it.
Both reads keep their stderr and carry a bound. A read that fails
silently is indistinguishable from a namespace with nothing to report,
which is the distinction being drawn; and the caller is blocked in
`wait`, so an unbounded read against a wedged apiserver would hold the
install open until the job's own timeout. The client budget is
strictly the smaller of the two so kubectl gets to name the reason
before the outer kill takes it away.
The wait helper is named for linstor because those diagnostics read
one namespace unconditionally: the wait is generic over its kubectl
arguments, what it dumps on the way out is not.
The new unit test runs the script end to end against a stub PATH
carrying a virtual clock, so a 15-minute budget elapses in a few
hundred forks and the timings are exact rather than wall-clock
approximate.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The post-install-prep node-list loop dialed the linstor-controller Service
(linstor+ssl://linstor-controller:3371) as soon as the controller Deployment
reported Available once. That gate goes stale: the controller carries
reloader.stakater.com/auto=true and mounts the cert-manager-issued API-TLS
Secret, so when cert-manager (re)writes that Secret during bring-up, reloader
rolls the single-replica controller. The roll drops the Service to zero ready
endpoints after Available already passed, and a dial into that window fails
with "[Errno 113] No route to host" (EHOSTUNREACH). The blind until-loop then
churned the CLI on that error instead of waiting for a routable controller.
Gate every probe on the linstor-controller Service having a ready endpoint, so
the CLI only dials a reachable controller and a mid-loop reload re-waits for
the endpoint instead of erroring. The "Online == 3" satellite-convergence
assertion and the dedicated 300s budget are preserved; a timeout now dumps
controller endpoints and pods for diagnostics before failing.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback on hack/e2e-post-install-prep.sh:31:
POSIX `wait` without arguments returns 0 even if a backgrounded
`kubectl exec` failed, which would mask partial LINSTOR prep failures
and let downstream tests run on a half-provisioned cluster. Collect
each background PID and `wait "$pid"` per job so `set -e` exits the
script on the first failed pool creation.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Two complementary changes that depend on the operator's
`--helmrelease-interval` flag landing via #2509:
1. Installer chart exposes `cozystackOperator.helmReleaseInterval`
(default empty -> flag not rendered -> operator default 5m
applies). E2E install passes `30s`. Eliminates the 8-10 min Flux
dead zone where dependency-blocked HRs (waiting on cert-manager
webhooks, CRDs) are requeued only every 5 min.
2. LINSTOR / StorageClass / MetalLB setup, previously sequential
`@test` cases after the 15 m platform HR wait, now run as a
background prep that completes during the platform reconcile.
None of them gates platform HR reconcile, so they can move out
of the critical path. Helper script `hack/e2e-post-install-prep.sh`
polls for its prerequisites and is awaited at the end of the
`Install Cozystack` test so failures still surface deterministically.
Includes the T2-race fix: wait on the LINSTOR HR being `Ready`
before polling for `deploy/linstor-controller`. The old 5-min hard
timeout fired during cold installs where the LINSTOR HR's dependency
chain takes longer than 5 min to resolve.
Production defaults are unchanged. The 30 s interval is opt-in per
install command.
Depends on:
- #2509 (operator-side `--helmrelease-interval` flag).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Aleksei Sviridkin