Files
cozystack/hack/sandbox-runner-headroom.bats
Myasnikov Daniil cc942fed68 test(tests): run unit suites with Bats and retain POSIX coverage
The shell translator rewrites test bodies and loses Bats reporting.
The unit lane needs complete TAP results and JUnit reports without
changing how live-cluster suites run.

Sourced production helpers still need a dash pass, and unit test bodies
need nounset without exporting shell options into subprocesses.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 11:04:00 +03:00

470 lines
22 KiB
Bash

#!/usr/bin/env bats
# Asserts that the e2e sandbox never fills a CI runner to its nominal capacity:
# every workflow job that boots the sandbox must run on a runner class with
# strictly more vCPUs and strictly more memory than the sandbox's QEMU guests
# demand.
#
# The sandbox boots a fixed set of QEMU management nodes, each with its own
# `-smp` and `-m`, and those resources are committed for the whole run. The
# host needs cycles and memory for its own work on top of that, so a 1:1
# commit leaves nothing to give it. That is an arithmetic statement about the
# configuration, and it is the entire basis for this check.
#
# Nothing enforced that pairing before this file: the guest demand lives in a
# bats file and the runner classes live in four workflows, so raising `-smp`
# or adding a lane on a smaller runner drifted them apart silently. That is how
# one lane came to sit on a 32-vCPU class while another ran the same sandbox on
# a 24-vCPU one.
#
# The scope is that drift and nothing more, and the limit is worth stating
# because the check is easy to over-read. It compares numbers. It attributes no
# red run to a runner class, predicts nothing about how fast a lane runs, and a
# green result means only that a lane is not oversubscribed on paper. Whether a
# lane is fast enough also depends on what else shares the host, which none of
# these files describe. Memory is compared because the same arithmetic applies
# to it and the numbers are already in the strings being read, not because a
# class that violates it exists today.
#
# Both sides are derived, never listed. The guest demand is read out of
# hack/e2e-prepare-cluster.bats by multiplying each qemu launch's `-smp` and
# `-m` by the iteration count of the `for` loop it sits in, so moving any of
# those moves this check with it. The job list is read out of .github/workflows
# by finding the jobs that invoke the `prepare-env` / `prepare-cluster` make
# targets, so a fifth lane written as an ordinary job is covered the day it is
# added rather than the day someone remembers to extend a literal list here.
#
# The floor of that derivation is the workflow file itself, and it is worth
# naming rather than leaving as an implied promise of total coverage. A lane
# that reached the sandbox through a composite action would carry the make
# target inside the action and only `runs-on` in the workflow, so neither the
# marker sweep nor the coverage cross-check would see it. Nothing in the tree
# is written that way today. A workflow reached through `workflow_call` is a
# different case and is covered, because the called file carries both its own
# `runs-on` and its own marker. Covering a composite action means teaching both
# legs to follow `uses:`, which is a larger change than widening a pattern.
#
# A derivation that cannot read its input has two ways out, and only one of
# them is safe. Guessing produces a number smaller than the truth, which passes
# a runner that cannot hold the sandbox; stopping produces a red naming the
# line. This one stops. The guest-side shapes in the list below are pinned by
# the fixture tests at the end of this file; the last bullet is a runner-side
# refusal, which only the live tree exercises.
#
# - a line naming qemu-system-x86_64 with no literal `-smp <n>` or no literal
# `-m <n>`, which covers `-smp cpus=8` and `-m size=24576` as well as an
# absent flag;
# - such a line outside any @test block, where no loop count applies to it;
# - a @test block still open at end of file, whose contents were never
# accounted for;
# - a `for` opener the strict pattern does not match, counted separately from
# the ones it does so that `do` on the following line is refused instead of
# silently multiplying by one;
# - a `for` list holding anything but literal words, so `$(seq 1 3)` is
# refused rather than counted as three whitespace tokens by coincidence;
# - a `while` or `until` loop in the same @test block as a qemu launch, since
# its iteration count is not a static property at all;
# - more than one qemu launch or more than one `for` loop in a single @test
# block;
# - a sandbox job whose `runs-on` does not carry an `oracle-vm-<n>cpu-<m>gb`
# class.
#
# Several of those are scoped to the @test block rather than to real nesting,
# and say so in their messages, because this file refuses rather than parses
# shell. A poll loop added beside a qemu launch trips the while/until refusal
# without enclosing anything; moving it into its own @test block is the answer,
# and the message says that rather than leaving the reader to derive it. The
# loop multiplier is block-scoped for the same reason: a strict `for` anywhere
# in the block multiplies the launch whether or not it encloses it. That
# over-counts rather than under-counts, so the worst it yields is a false red,
# and the one-loop-per-block refusal bounds how far it can go.
#
# The qemu match is deliberately any line naming `qemu-system-x86_64`, which is
# wider than a launch. Narrowing it to lines that begin with the binary would
# skip a launch written as `sudo qemu-system-x86_64` or with a leading `env`,
# and a skipped launch lowers the demand this compares, which is the one
# direction that must never happen quietly. So the width stays and a line that
# names the binary without a literal `-smp`/`-m` is refused by name.
#
# Unit note for the memory leg: qemu's `-m` is MiB, and the runner class names
# its memory in `gb`. This reads that `gb` as GiB, which is the permissive
# direction, not the safe one: if the vendor means GB, the runner holds less
# than assumed and this leg lets through a class it should not. It is written
# that way because the inequality it asserts holds under either reading for the
# classes in .github/actionlint.yaml, so the looser unit costs nothing today.
#
# The coverage cross-check exists for the other rot direction. Matching jobs on
# a make-target name says nothing when the name changes or the YAML is
# reindented: the parser simply finds fewer jobs and the survivors still pass.
# So every workflow file that drives the sandbox make targets at all must yield
# at least one matched job, which turns a half-broken parser red instead of
# quietly narrowing what is checked.
#
# Harness compatibility note: CI runs this file under Bats through
# `make bats-unit-tests`. It also remains compatible with the narrower legacy
# `hack/cozytest.sh` translator, so it avoids `run`, `$status`, `$output`,
# `skip`, and setup()/teardown(). A `!`-negated command never trips errexit, so
# every assertion below is written as an explicit `if ... return 1`. The legacy
# translator injects `return 0` before a top-level helper's closing brace, so
# nothing is factored into one. It also rewrites bare `}` lines and lines that
# begin with `@test `, so the fixtures below use `printf` instead of heredocs.
# Paths stay repo-root-relative.
#
# Run with: bats hack/sandbox-runner-headroom.bats
load test_helper
# The guest-side derivation, held in one place so the check that runs against
# the live tree and the fixtures that pin its refusals cannot drift apart.
# Prints the vCPU total then the MiB total on success, one ERROR line
# otherwise. `exit` inside an awk rule still runs END, so the error path raises
# a flag END reads first: stdout is the only channel that survives to a caller.
GUEST_DEMAND_AWK='
/^@test / {
if (inblk) { print "ERROR unclosed @test block above line " NR; failed = 1; exit 1 }
inblk = 1; nfor = 0; nforraw = 0; nloop = 0; forcount = 0
nqemu = 0; smp = ""; mem = ""; baditem = ""
next
}
inblk && /^}/ {
if (nqemu > 0) {
blk = " in the @test block ending at line " NR
if (nqemu > 1) { print "ERROR " nqemu " lines naming qemu-system-x86_64" blk; failed = 1; exit 1 }
if (smp == "") { print "ERROR line naming qemu-system-x86_64 with no literal -smp <n>" blk; failed = 1; exit 1 }
if (mem == "") { print "ERROR line naming qemu-system-x86_64 with no literal -m <n>" blk; failed = 1; exit 1 }
if (nloop > 0) { print "ERROR while/until loop in the same @test block as a qemu launch" blk; failed = 1; exit 1 }
if (nforraw > nfor) { print "ERROR for opener the strict pattern could not read" blk; failed = 1; exit 1 }
if (nfor > 1) { print "ERROR " nfor " for-loops" blk; failed = 1; exit 1 }
if (baditem != "") { print "ERROR non-literal item \"" baditem "\" in the for list" blk; failed = 1; exit 1 }
mult = (nfor == 1 ? forcount : 1)
total_cpu += smp * mult
total_mem += mem * mult
launches += 1
}
inblk = 0
next
}
!inblk && /qemu-system-x86_64/ {
print "ERROR line naming qemu-system-x86_64 outside any @test block at line " NR
failed = 1
exit 1
}
inblk {
line = $0
if (line ~ /^[[:space:]]*(while|until)[[:space:]]/) nloop += 1
if (line ~ /^[[:space:]]*for[[:space:]]/) {
nforraw += 1
if (line ~ /^[[:space:]]*for[[:space:]]+[A-Za-z_][A-Za-z0-9_]*[[:space:]]+in[[:space:]]+[^;]+;[[:space:]]*do[[:space:]]*$/) {
sub(/^[[:space:]]*for[[:space:]]+[A-Za-z_][A-Za-z0-9_]*[[:space:]]+in[[:space:]]+/, "", line)
sub(/;[[:space:]]*do[[:space:]]*$/, "", line)
sub(/^[[:space:]]+/, "", line)
sub(/[[:space:]]+$/, "", line)
nfor += 1
forcount = split(line, items, /[[:space:]]+/)
for (k = 1; k <= forcount; k++) {
if (items[k] !~ /^[A-Za-z0-9_.-]+$/) baditem = items[k]
}
}
line = $0
}
if (line ~ /qemu-system-x86_64/) {
nqemu += 1
if (match(line, /-smp[[:space:]]+[0-9]+([[:space:]]|$)/)) {
smp = substr(line, RSTART, RLENGTH)
sub(/-smp[[:space:]]+/, "", smp)
sub(/[^0-9]/, "", smp)
}
if (match(line, /-m[[:space:]]+[0-9]+([[:space:]]|$)/)) {
mem = substr(line, RSTART, RLENGTH)
sub(/-m[[:space:]]+/, "", mem)
sub(/[^0-9]/, "", mem)
}
}
}
END {
if (failed) exit 1
if (inblk) { print "ERROR @test block still open at end of file"; exit 1 }
if (launches == 0) { print "ERROR no qemu-system-x86_64 launch found"; exit 1 }
print total_cpu
print total_mem
}
'
@test "every workflow job that boots the e2e sandbox leaves spare host capacity" {
# Read from the QEMU sandbox file for every lane, container lanes included:
# the `prepare-env` substring also matches `prepare-env-container`, and the
# compose file gives its three nodes the same 8 CPU / 24576 MiB the QEMU
# guests get, by design (packages/core/testing/Makefile). So the figures are
# right for both substrates, but only because the two agree -- and that they
# agree is itself pinned, in hack/container-lane-capacity_test.bats, which
# holds hack/e2e-compose.yaml to the same literals. If the container lane is
# ever sized differently, this test has to start reading e2e-compose.yaml for
# those jobs; it will not notice on its own.
sandbox_file=hack/e2e-prepare-cluster.bats
workflow_dir=.github/workflows
# `|| true` because the assignment inherits awk's status, and under errexit a
# refusal would kill the test right here, before the branch below can print
# the sentence that says which shapes the parser needs.
guest=$(awk "$GUEST_DEMAND_AWK" "$sandbox_file") || true
guest_vcpus=$(printf '%s\n' "$guest" | sed -n 1p)
guest_mib=$(printf '%s\n' "$guest" | sed -n 2p)
for value in "$guest_vcpus" "$guest_mib"; do
case "$value" in
""|*[!0-9]*)
echo "could not read the sandbox guest demand from $sandbox_file:" >&2
printf '%s\n' "$guest" >&2
echo "every qemu-system-x86_64 launch must carry a literal '-smp <n>' and '-m <n>', sit inside a closed @test block, and be enclosed by at most one literal 'for VAR in ...; do' loop." >&2
return 1
;;
esac
done
if [ "$guest_vcpus" -le 0 ] || [ "$guest_mib" -le 0 ]; then
echo "the sandbox guest demand read from $sandbox_file is $guest_vcpus vCPU / $guest_mib MiB" >&2
return 1
fi
# ── Runner side. One line per job that drives the sandbox make targets:
# workflow path, job name, the vCPU and GiB of its runner class (or NONE for
# both) and the raw runs-on value. Comment lines are dropped before the
# marker match, so prose naming e2e-prepare-cluster.bats above a job cannot
# enrol it. Both globs are passed unquoted: if either matches nothing awk
# gets a literal path and exits 2, which errexit turns into a red without the
# named line the refusals above carry. Louder than they are, and still the
# safe direction, since no workflow extension drops out of the sweep quietly.
sandbox_jobs=$(awk '
function class_of(s, n) {
if (match(s, /oracle-vm-[0-9]+cpu-[0-9]+gb/)) {
n = substr(s, RSTART, RLENGTH)
sub(/^oracle-vm-/, "", n)
sub(/cpu-/, " ", n)
sub(/gb$/, "", n)
return n
}
return "NONE NONE"
}
function flush() {
if (job != "" && sandbox) print wf, job, class_of(ro), ro
job = ""; ro = ""; sandbox = 0
}
FNR == 1 { flush(); wf = FILENAME }
/^ [A-Za-z0-9_-]+:[[:space:]]*$/ {
flush()
job = $0
sub(/^[[:space:]]+/, "", job)
sub(/:[[:space:]]*$/, "", job)
next
}
job != "" {
if ($0 ~ /^[[:space:]]*#/) next
if ($0 ~ /^ runs-on:/) {
ro = $0
sub(/^[[:space:]]*runs-on:[[:space:]]*/, "", ro)
}
if ($0 ~ /prepare-env/ || $0 ~ /prepare-cluster/) sandbox = 1
}
END { flush() }
' "$workflow_dir"/*.yaml "$workflow_dir"/*.yml)
if [ -z "$sandbox_jobs" ]; then
echo "no job in $workflow_dir invokes the sandbox make targets" >&2
echo "this check matches jobs on 'prepare-env' / 'prepare-cluster'; if those targets were renamed, rename them here too." >&2
return 1
fi
# ── Coverage cross-check, before the comparison it guards. A file that runs
# anything under packages/core/testing is a sandbox lane, so failing to match
# a job in it means the parser above lost the file, not that the file is
# innocent.
for wf in "$workflow_dir"/*.yaml "$workflow_dir"/*.yml; do
[ -e "$wf" ] || continue
if grep -v '^[[:space:]]*#' "$wf" | grep -q 'packages/core/testing'; then
if printf '%s\n' "$sandbox_jobs" | grep -qF "$wf "; then
continue
fi
# Named exemption: the isp-hosted lane starts the sandbox container but
# no guests -- its cluster is kind, sized by the runner itself -- so it
# has no guest demand to compare. It is checked after the job match, so
# the day that lane calls prepare-env or prepare-cluster it is sized
# like any other.
case "$wf" in
"$workflow_dir"/e2e-isp-hosted.yaml) continue ;;
esac
echo "$wf runs the packages/core/testing make targets but no job in it matched the sandbox marker" >&2
echo "the job parser reads job names at two-space indent and runs-on at four; a reindent or a renamed target breaks it silently." >&2
return 1
fi
done
violations=
while read -r wf job cpus gib raw; do
if [ "$cpus" = "NONE" ]; then
echo "job '$job' in $wf boots the e2e sandbox but its runs-on names no oracle-vm-<n>cpu-<m>gb class: ${raw:-<empty>}" >&2
echo "this check cannot size a runner it cannot name, and guessing one would be worse than stopping." >&2
return 1
fi
runner_mib=$((gib * 1024))
if [ "$guest_vcpus" -ge "$cpus" ]; then
violations="$violations
$wf job '$job': $guest_vcpus guest vCPU on a ${cpus}-vCPU runner ($raw)"
fi
if [ "$guest_mib" -ge "$runner_mib" ]; then
violations="$violations
$wf job '$job': $guest_mib MiB of guest memory on a ${gib}-GiB runner ($raw)"
fi
done <<EOF
$sandbox_jobs
EOF
if [ -n "$violations" ]; then
echo "the e2e sandbox demands $guest_vcpus guest vCPU and $guest_mib MiB (from $sandbox_file)," >&2
echo "which leaves no host headroom on these runners:" >&2
printf '%s\n' "$violations" >&2
echo "Move the job to a larger runner class, or lower the sandbox's -smp / -m." >&2
return 1
fi
}
# ── Fixtures for the guest-side derivation.
#
# The check above passes on the live tree whether or not the refusals still
# work, so loosening the awk would be a silent no-op without these. Each one
# feeds a synthetic file to the same program the check uses and asserts on the
# value it prints.
@test "guest demand multiplies -smp and -m by the loop count" {
fixdir=$(mktemp -d)
printf '@test "boot" {\n for i in 1 2 3; do\n qemu-system-x86_64 -smp 8 -m 24576 -display none\n done\n}\n' > "$fixdir/f.bats"
demand=$(awk "$GUEST_DEMAND_AWK" "$fixdir/f.bats" 2>&1) || true
rm -rf "$fixdir"
cpu=$(printf '%s\n' "$demand" | sed -n 1p)
mib=$(printf '%s\n' "$demand" | sed -n 2p)
if [ "$cpu" != 24 ] || [ "$mib" != 73728 ]; then
echo "expected 24 vCPU / 73728 MiB, got: [$demand]" >&2
return 1
fi
}
@test "guest demand counts a launch outside any loop exactly once" {
fixdir=$(mktemp -d)
printf '@test "boot" {\n qemu-system-x86_64 -smp 8 -m 24576 -display none\n}\n' > "$fixdir/f.bats"
demand=$(awk "$GUEST_DEMAND_AWK" "$fixdir/f.bats" 2>&1) || true
rm -rf "$fixdir"
cpu=$(printf '%s\n' "$demand" | sed -n 1p)
mib=$(printf '%s\n' "$demand" | sed -n 2p)
if [ "$cpu" != 8 ] || [ "$mib" != 24576 ]; then
echo "expected 8 vCPU / 24576 MiB, got: [$demand]" >&2
return 1
fi
}
@test "guest demand refuses a qemu line outside any @test block" {
fixdir=$(mktemp -d)
printf 'boot_node() {\n qemu-system-x86_64 -smp 4 -m 8192\n}\n@test "boot" {\n for i in 1 2 3; do\n qemu-system-x86_64 -smp 8 -m 24576\n done\n}\n' > "$fixdir/f.bats"
demand=$(awk "$GUEST_DEMAND_AWK" "$fixdir/f.bats" 2>&1) || true
rm -rf "$fixdir"
case "$demand" in
ERROR*outside\ any\ @test\ block*) : ;;
*) echo "expected a refusal naming the out-of-block line, got: [$demand]" >&2; return 1 ;;
esac
}
@test "guest demand refuses a @test block left open at end of file" {
fixdir=$(mktemp -d)
printf '@test "boot" {\n for i in 1 2 3; do\n qemu-system-x86_64 -smp 8 -m 24576\n done\n' > "$fixdir/f.bats"
demand=$(awk "$GUEST_DEMAND_AWK" "$fixdir/f.bats" 2>&1) || true
rm -rf "$fixdir"
case "$demand" in
ERROR*still\ open\ at\ end\ of\ file*) : ;;
*) echo "expected a refusal naming the unclosed block, got: [$demand]" >&2; return 1 ;;
esac
}
@test "guest demand refuses a launch with no literal -smp" {
fixdir=$(mktemp -d)
printf '@test "boot" {\n for i in 1 2 3; do\n qemu-system-x86_64 -smp cpus=8 -m 24576\n done\n}\n' > "$fixdir/f.bats"
demand=$(awk "$GUEST_DEMAND_AWK" "$fixdir/f.bats" 2>&1) || true
rm -rf "$fixdir"
case "$demand" in
ERROR*no\ literal\ -smp*) : ;;
*) echo "expected a refusal naming -smp, got: [$demand]" >&2; return 1 ;;
esac
}
@test "guest demand refuses a launch with no literal -m" {
fixdir=$(mktemp -d)
printf '@test "boot" {\n for i in 1 2 3; do\n qemu-system-x86_64 -smp 8 -m size=24576\n done\n}\n' > "$fixdir/f.bats"
demand=$(awk "$GUEST_DEMAND_AWK" "$fixdir/f.bats" 2>&1) || true
rm -rf "$fixdir"
case "$demand" in
ERROR*no\ literal\ -m*) : ;;
*) echo "expected a refusal naming -m, got: [$demand]" >&2; return 1 ;;
esac
}
@test "guest demand refuses a for opener the strict pattern cannot read" {
fixdir=$(mktemp -d)
printf '@test "boot" {\n for i in 1 2 3\n do\n qemu-system-x86_64 -smp 8 -m 24576\n done\n}\n' > "$fixdir/f.bats"
demand=$(awk "$GUEST_DEMAND_AWK" "$fixdir/f.bats" 2>&1) || true
rm -rf "$fixdir"
case "$demand" in
ERROR*for\ opener*) : ;;
*) echo "expected a refusal naming the for opener, got: [$demand]" >&2; return 1 ;;
esac
}
@test "guest demand refuses a for list that is not literal words" {
fixdir=$(mktemp -d)
printf '@test "boot" {\n for i in $(seq 1 3); do\n qemu-system-x86_64 -smp 8 -m 24576\n done\n}\n' > "$fixdir/f.bats"
demand=$(awk "$GUEST_DEMAND_AWK" "$fixdir/f.bats" 2>&1) || true
rm -rf "$fixdir"
case "$demand" in
ERROR*non-literal\ item*) : ;;
*) echo "expected a refusal naming the non-literal item, got: [$demand]" >&2; return 1 ;;
esac
}
@test "guest demand refuses a while loop beside a qemu launch" {
fixdir=$(mktemp -d)
printf '@test "boot" {\n i=1\n while [ $i -le 3 ]; do\n qemu-system-x86_64 -smp 8 -m 24576\n done\n}\n' > "$fixdir/f.bats"
demand=$(awk "$GUEST_DEMAND_AWK" "$fixdir/f.bats" 2>&1) || true
rm -rf "$fixdir"
case "$demand" in
ERROR*while/until\ loop*) : ;;
*) echo "expected a refusal naming the while loop, got: [$demand]" >&2; return 1 ;;
esac
}
@test "guest demand refuses two qemu lines in one @test block" {
fixdir=$(mktemp -d)
printf '@test "boot" {\n for i in 1 2 3; do\n qemu-system-x86_64 -smp 8 -m 24576\n qemu-system-x86_64 -smp 4 -m 8192\n done\n}\n' > "$fixdir/f.bats"
demand=$(awk "$GUEST_DEMAND_AWK" "$fixdir/f.bats" 2>&1) || true
rm -rf "$fixdir"
case "$demand" in
ERROR*lines\ naming\ qemu-system-x86_64*) : ;;
*) echo "expected a refusal naming the second launch, got: [$demand]" >&2; return 1 ;;
esac
}
@test "guest demand refuses two for loops in one @test block" {
fixdir=$(mktemp -d)
printf '@test "boot" {\n for j in a b; do\n echo "$j"\n done\n for i in 1 2 3; do\n qemu-system-x86_64 -smp 8 -m 24576\n done\n}\n' > "$fixdir/f.bats"
demand=$(awk "$GUEST_DEMAND_AWK" "$fixdir/f.bats" 2>&1) || true
rm -rf "$fixdir"
case "$demand" in
ERROR*for-loops*) : ;;
*) echo "expected a refusal naming the second loop, got: [$demand]" >&2; return 1 ;;
esac
}
@test "guest demand refuses a file with no qemu launch at all" {
fixdir=$(mktemp -d)
printf '@test "nothing" {\n true\n}\n' > "$fixdir/f.bats"
demand=$(awk "$GUEST_DEMAND_AWK" "$fixdir/f.bats" 2>&1) || true
rm -rf "$fixdir"
case "$demand" in
ERROR*no\ qemu-system-x86_64\ launch\ found*) : ;;
*) echo "expected a refusal naming the absent launch, got: [$demand]" >&2; return 1 ;;
esac
}