Commit Graph
6 Commits
Author SHA1 Message Date
Myasnikov Daniil cc942fed68 test(tests): run unit suites with Bats and retain POSIX coverage
The shell translator rewrites test bodies and loses Bats reporting.
The unit lane needs complete TAP results and JUnit reports without
changing how live-cluster suites run.

Sourced production helpers still need a dash pass, and unit test bodies
need nounset without exporting shell options into subprocesses.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 11:04:00 +03:00
Aleksei Sviridkin cb7a579065 fix(e2e): compare LINSTOR pool datasets, not free bytes, at cleanup
For a node with only thick ZFS pools, `linstor sp l` reports the free
capacity cached at the satellite's last report rather than asking the
satellite, and that report can land while `zfs destroy` is still
freeing blocks in the background. A cleanup that had removed every
volume it created could then wait out its whole budget against a
figure that never moved. That residue has reached about 0.9 GiB, with
no known upper bound, against 1 GiB for the smallest volume a suite is
known to have leaked, so a byte tolerance has no safe value between
the two.

Record the datasets under each node's pool before the suite, read with
`zfs list` on that node's satellite, and wait until no dataset outside
that set remains. A leaked volume is then reported by name and node,
and a destroy still freeing space in the background does not count.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 02:17:00 +03:00
Aleksei Sviridkin 64b3c9619f fix(e2e): hold replicated-lane LINSTOR cleanup to a pre-suite baseline
The kubernetes suites' cleanup in the replicated lane demanded 90 GiB
free on every LINSTOR node. Volumes that earlier suites and the
platform legitimately keep can hold a node under that figure even when
the suite reclaimed everything it used, so the cleanup failed suites
that had leaked nothing.

Both lanes now record the per-node baseline before the suite creates
anything and compare against it on cleanup. The 90 GiB figure is what
a replicated suite needs to place its workers and CDI scratch volumes,
so it stays as a precondition checked before the baseline, and a
shortfall there is reported as the sandbox's state rather than the
suite's.

A run that fails before recording its baseline used to leave the
previous suite's file in place for its cleanup to compare against; the
file is now removed when the suite starts.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 15:27:18 +03:00
Aleksei Sviridkin ff2a677fc0 fix(e2e): say why the MetalLB CRD and CDI StorageProfile waits failed
Both waits ran as `timeout N sh -ec 'until kubectl get ...'` with the
read's output discarded. On elapse `timeout` exits 124 and `set -e`
ends the script, so the last line in the log is the one announcing the
wait: CRDs that are late and an operator that never got far enough to
install them read the same.

A shared deadline loop now polls quietly and, on the way out, reads the
objects once more with stderr kept, so the log names which object is
missing. The MetalLB wait also lists the metallb HelmRelease, whose
state separates those two cases. The loop runs on the script's own
clock, which lets the unit suite drive it with the virtual one.

The unit suite's runaway guard in the sleep stub now kills its caller
as well, because a failing exit status does not stop a loop that runs
where set -e is off, and a guard that cannot stop the loop turns a
broken deadline into a hung test.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 00:58:08 +03:00
Aleksei Sviridkin d07099f13c fix(e2e): bound the calls that do work in post-install-prep
The install step blocks in `wait` on this script, so a kubectl call
that never returns does not fail the step: it holds it open until the
job's own ceiling cancels the run, which leaves different evidence
behind than a failed test. Only the waits carried a budget. The execs
that create storage pools, the applies, the CDI patch, and the reads
inside the LINSTOR node loop did not, and the loop's deadline is
checked between iterations, on the other side of a call that hangs.

The reads that explain a node-loop timeout were unbounded too, on the
one path that runs only when the cluster is already unhealthy.

Each call now runs under an outer timeout, and plain requests carry a
client timeout strictly inside it so kubectl names the reason before
the kill lands. The helpers take the bound as a prefix that stays empty
when unit tests source them, since `timeout` would bypass the kubectl
function those tests substitute.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 00:58:07 +03:00
Myasnikov Daniil 186547d835 test(e2e): run the merge-gating lanes on Talos containers
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.

The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.

Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.

Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.

The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.

Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.

One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.

Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-05 13:19:17 +05:00