Commit Graph
6 Commits
Author SHA1 Message Date
Myasnikov Daniil 186547d835 test(e2e): run the merge-gating lanes on Talos containers
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.

The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.

Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.

Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.

The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.

Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.

One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.

Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-05 13:19:17 +05:00
Aleksei Sviridkin 86674c7084 fix(e2e): budget each linstor wait separately in post-install-prep
The post-install prep script waits out a serial reconcile chain --
the linstor HelmRelease, then the linstor-controller Deployment --
against a single 15-minute deadline anchored at the script's start.
The head of the chain therefore eats the window and the tail inherits
whatever remains, so an install fails whenever the chain merely runs
long rather than never converging.

Measured on one such failure: the HelmRelease went Ready 452s in, and
the Deployment needed a further ~520s because the controller's DB
migration lost its apiserver connection and retried four times before
succeeding. Neither figure exceeds 900s on its own; only their sum
did, and the install reported the Deployment as though it were broken.

Give each wait its own budget, started when the link before it
completed -- the shape the LINSTOR node and MetalLB CRD waits further
down already have. What that buys is not speed but a guarantee: every
link gets its whole allowance regardless of what the links ahead of
it spent, so the tail can no longer be failed by the head being slow.

It is a trade, and the other half is worth stating. The script's worst
case becomes 900 + 900 + 300 + 300 = 40m against the 180m the job
carries, where before the two waits shared one 15m window. And a link
that is genuinely stuck is reported later rather than sooner: a
Deployment that never converges surfaces at 1355s where the shared
window surfaced it at 900s, on that run's timings. A HelmRelease that
never becomes Ready surfaces at the same moment as before, since its
budget still starts with the script.

A timeout now also lists the linstor pods and the namespace events.
The description names the object waited on, not the link that held it
up, and the same "Deployment not Available" has been reached both with
the controller pod crash-looping its migration init container and with
that pod blocked on a client TLS Secret cert-manager had not issued
yet. Telling those apart previously required the run's bundle.

The events come from `kubectl events` rather than `kubectl get
events`. It orders by when an event was last seen, so a condition
repeating for minutes -- what a stuck mount looks like -- stays inside
the window instead of being pushed out by newer one-off events, and it
carries no --sort-by, so it cannot fail the whole read the way a sort
key does when one item lacks it.

Both reads keep their stderr and carry a bound. A read that fails
silently is indistinguishable from a namespace with nothing to report,
which is the distinction being drawn; and the caller is blocked in
`wait`, so an unbounded read against a wedged apiserver would hold the
install open until the job's own timeout. The client budget is
strictly the smaller of the two so kubectl gets to name the reason
before the outer kill takes it away.

The wait helper is named for linstor because those diagnostics read
one namespace unconditionally: the wait is generic over its kubectl
arguments, what it dumps on the way out is not.

The new unit test runs the script end to end against a stub PATH
carrying a virtual clock, so a 15-minute budget elapses in a few
hundred forks and the timings are exact rather than wall-clock
approximate.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-10 00:10:53 +03:00
Aleksei Sviridkin 41eb6f912b fix(e2e): gate LINSTOR node-list probe on a ready controller endpoint
The post-install-prep node-list loop dialed the linstor-controller Service
(linstor+ssl://linstor-controller:3371) as soon as the controller Deployment
reported Available once. That gate goes stale: the controller carries
reloader.stakater.com/auto=true and mounts the cert-manager-issued API-TLS
Secret, so when cert-manager (re)writes that Secret during bring-up, reloader
rolls the single-replica controller. The roll drops the Service to zero ready
endpoints after Available already passed, and a dial into that window fails
with "[Errno 113] No route to host" (EHOSTUNREACH). The blind until-loop then
churned the CLI on that error instead of waiting for a routable controller.

Gate every probe on the linstor-controller Service having a ready endpoint, so
the CLI only dials a reachable controller and a mid-loop reload re-waits for
the endpoint instead of erroring. The "Online == 3" satellite-convergence
assertion and the dedicated 300s budget are preserved; a timeout now dumps
controller endpoints and pods for diagnostics before failing.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-09 00:44:34 +03:00
Myasnikov DaniilandClaude Opus 4.8 c1aca89fde fix(e2e): drive LINSTOR post-install waits off a single 15m deadline (#2928)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-18 10:40:06 +05:00
Myasnikov Daniil d8da80bdd2 fix(e2e): propagate failures from parallel LINSTOR pool jobs
Address review feedback on hack/e2e-post-install-prep.sh:31:
POSIX `wait` without arguments returns 0 even if a backgrounded
`kubectl exec` failed, which would mask partial LINSTOR prep failures
and let downstream tests run on a half-provisioned cluster. Collect
each background PID and `wait "$pid"` per job so `set -e` exits the
script on the first failed pool creation.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-06 12:24:01 +05:00
Myasnikov Daniil 7ed6444108 feat(installer,e2e): cozystackOperator.helmReleaseInterval value + parallel post-install prep
Two complementary changes that depend on the operator's
`--helmrelease-interval` flag landing via #2509:

1. Installer chart exposes `cozystackOperator.helmReleaseInterval`
   (default empty -> flag not rendered -> operator default 5m
   applies). E2E install passes `30s`. Eliminates the 8-10 min Flux
   dead zone where dependency-blocked HRs (waiting on cert-manager
   webhooks, CRDs) are requeued only every 5 min.

2. LINSTOR / StorageClass / MetalLB setup, previously sequential
   `@test` cases after the 15 m platform HR wait, now run as a
   background prep that completes during the platform reconcile.
   None of them gates platform HR reconcile, so they can move out
   of the critical path. Helper script `hack/e2e-post-install-prep.sh`
   polls for its prerequisites and is awaited at the end of the
   `Install Cozystack` test so failures still surface deterministically.

Includes the T2-race fix: wait on the LINSTOR HR being `Ready`
before polling for `deploy/linstor-controller`. The old 5-min hard
timeout fired during cold installs where the LINSTOR HR's dependency
chain takes longer than 5 min to resolve.

Production defaults are unchanged. The 30 s interval is opt-in per
install command.

Depends on:
- #2509 (operator-side `--helmrelease-interval` flag).

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
2026-05-05 14:41:24 +05:00