Files
cozystack/hack/linstor-storage-mode_test.bats
T
Myasnikov Daniil 186547d835 test(e2e): run the merge-gating lanes on Talos containers
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.

The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.

Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.

Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.

The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.

Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.

One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.

Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-05 13:19:17 +05:00

314 lines
13 KiB
Bash

#!/usr/bin/env bats
# Unit coverage for the lane-specific E2E storage contract: LINSTOR pool
# registration, management StorageClasses, and downstream Chainsaw fixtures.
# The library guard keeps this suite cluster-free under hack/cozytest.sh.
HACK_DIR="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")" && pwd)"
POST_PREP="$HACK_DIR/e2e-post-install-prep.sh"
# shellcheck source=/dev/null
E2E_POST_INSTALL_PREP_LIB=true . "$POST_PREP"
kubectl() { printf '%s\n' "$*"; }
@test "QEMU mode creates data from the satellite block device" {
unset COZY_LINSTOR_DRBD_ENABLED
command=$(create_linstor_storage_pool srv2)
expected='exec -n cozy-linstor deploy/linstor-controller -- linstor physical-storage create-device-pool zfs srv2 /dev/vdc --pool-name data --storage-pool data'
if [ "$command" != "$expected" ]; then
echo "unexpected QEMU pool command: $command" >&2
return 1
fi
}
@test "container mode registers the node-specific pre-created zpool" {
COZY_LINSTOR_DRBD_ENABLED=false
command=$(create_linstor_storage_pool srv2)
expected='exec -n cozy-linstor deploy/linstor-controller -- linstor storage-pool create zfs srv2 data data-srv2'
if [ "$command" != "$expected" ]; then
echo "unexpected container pool command: $command" >&2
return 1
fi
}
@test "QEMU mode renders local and replicated StorageClasses" {
unset COZY_LINSTOR_DRBD_ENABLED
manifest=$(render_linstor_storageclasses)
names=$(printf '%s\n' "$manifest" | yq -N '.metadata.name')
expected_names=$(printf '%s\n' local replicated)
replicated_layers=$(printf '%s\n' "$manifest" | yq 'select(.metadata.name == "replicated") | .parameters."linstor.csi.linbit.com/layerList"')
if [ "$names" != "$expected_names" ]; then
echo "unexpected QEMU StorageClasses: $names" >&2
return 1
fi
if [ "$replicated_layers" != "drbd storage" ]; then
echo "replicated StorageClass lost its DRBD layer: $replicated_layers" >&2
return 1
fi
}
@test "container mode renders only the production local StorageClass" {
COZY_LINSTOR_DRBD_ENABLED=false
manifest=$(render_linstor_storageclasses)
names=$(printf '%s\n' "$manifest" | yq -N '.metadata.name')
local_layers=$(printf '%s\n' "$manifest" | yq '.parameters."linstor.csi.linbit.com/layerList"')
remote=$(printf '%s\n' "$manifest" | yq '.parameters."linstor.csi.linbit.com/allowRemoteVolumeAccess"')
if [ "$names" != "local" ]; then
echo "container mode emitted unexpected StorageClasses: $names" >&2
return 1
fi
if [ "$local_layers" != "storage" ] || [ "$remote" != "false" ]; then
echo "container local StorageClass has unexpected parameters: layerList=$local_layers allowRemoteVolumeAccess=$remote" >&2
return 1
fi
}
@test "container mode pins CDI local StorageProfile to RWO Block" {
command=$(patch_local_cdi_storage_profile)
expected='patch storageprofile local --type merge -p {"spec":{"claimPropertySets":[{"accessModes":["ReadWriteOnce"],"volumeMode":"Block"}]}}'
if [ "$command" != "$expected" ]; then
echo "unexpected CDI StorageProfile patch: $command" >&2
return 1
fi
if ! grep -Fq 'timeout 600 sh -ec '\''until kubectl get storageprofile local' "$POST_PREP"; then
echo "container prep does not wait for CDI to create StorageProfile/local" >&2
return 1
fi
}
@test "invalid DRBD mode is rejected before cluster access" {
COZY_LINSTOR_DRBD_ENABLED=unsupported
if validate_linstor_storage_mode; then
echo "invalid DRBD mode unexpectedly passed validation" >&2
return 1
fi
}
@test "tenant Kubernetes storage defaults to replicated and accepts local only explicitly" {
# Sourcing inside the test keeps run-kubernetes.sh's live-cluster
# cozy_cleanup helper out of cozytest.sh's file-level cleanup discovery.
# shellcheck source=/dev/null
. "$HACK_DIR/e2e-chainsaw/_lib/run-kubernetes.sh"
unset COZY_E2E_STORAGE_CLASS
default_class=$(cozy_e2e_storage_class)
COZY_E2E_STORAGE_CLASS=local
container_class=$(cozy_e2e_storage_class)
if [ "$default_class" != replicated ] || [ "$container_class" != local ]; then
echo "unexpected tenant storage classes: default=$default_class container=$container_class" >&2
return 1
fi
COZY_E2E_STORAGE_CLASS=unsupported
if cozy_e2e_storage_class; then
echo "invalid tenant storage class unexpectedly passed validation" >&2
return 1
fi
}
@test "local storage uses its saved LINSTOR pool baseline" {
# shellcheck source=/dev/null
. "$HACK_DIR/e2e-chainsaw/_lib/run-kubernetes.sh"
COZY_E2E_STORAGE_CLASS=local
cozy_wait_linstor_pool_free() { printf 'unexpected pool wait\n'; return 99; }
cozy_wait_linstor_pool_baseline() { printf 'baseline-wait:%s:%s\n' "$1" "$2"; }
output=$(cozy_wait_linstor_pool_reclaimed 90 300)
if printf '%s\n' "$output" | grep -Fq 'unexpected pool wait'; then
echo "local storage entered the replicated-capacity wait" >&2
return 1
fi
[ "$output" = 'baseline-wait:300:524288' ]
}
@test "replicated storage retains the LINSTOR free-capacity barrier" {
# shellcheck source=/dev/null
. "$HACK_DIR/e2e-chainsaw/_lib/run-kubernetes.sh"
COZY_E2E_STORAGE_CLASS=replicated
cozy_wait_linstor_pool_free() { printf 'pool-wait:%s:%s\n' "$1" "$2"; }
output=$(cozy_wait_linstor_pool_reclaimed 91 17)
[ "$output" = 'pool-wait:91:17' ]
}
@test "local pool baseline comparison is per-node and tolerates only metadata drift" {
# shellcheck source=/dev/null
. "$HACK_DIR/e2e-chainsaw/_lib/run-kubernetes.sh"
baseline=$(printf 'srv1:67108864\nsrv2:20787200\nsrv3:60817408')
within_tolerance=$(printf 'srv1:67108864\nsrv2:20262912\nsrv3:60817408')
leaked_worker=$(printf 'srv1:67108864\nsrv2:20787200\nsrv3:39845888')
missing_node=$(printf 'srv1:67108864\nsrv3:60817408')
cozy_linstor_pools_at_baseline "$baseline" "$within_tolerance" 524288
if cozy_linstor_pools_at_baseline "$baseline" "$leaked_worker" 524288; then
echo "a 20 GiB worker leak passed the local-pool baseline" >&2
return 1
fi
if cozy_linstor_pools_at_baseline "$baseline" "$missing_node" 524288; then
echo "a missing LINSTOR node passed the local-pool baseline" >&2
return 1
fi
}
@test "local pool baseline capture persists all three numeric satellite rows" {
# shellcheck source=/dev/null
. "$HACK_DIR/e2e-chainsaw/_lib/run-kubernetes.sh"
COZY_LINSTOR_POOL_BASELINE_FILE="_out/tmp/linstor-baseline-$$"
kubectl() { printf 'srv3:60817408\nsrv1:67108864\nsrv2:20787200\n'; }
output=$(cozy_capture_linstor_pool_baseline)
captured=$(cat "$COZY_LINSTOR_POOL_BASELINE_FILE")
rm -f "$COZY_LINSTOR_POOL_BASELINE_FILE"
[ "$captured" = "$(printf 'srv3:60817408\nsrv1:67108864\nsrv2:20787200')" ]
printf '%s\n' "$output" | grep -Fq 'local LINSTOR pool baseline recorded'
}
@test "Kubernetes cleanup fails when local LINSTOR capacity is not reclaimed" {
# shellcheck source=/dev/null
. "$HACK_DIR/e2e-chainsaw/_lib/run-kubernetes.sh"
kubectl() { return 0; }
cozy_wait_tenant_drained() { return 0; }
cozy_wait_linstor_pool_reclaimed() { return 42; }
if cozy_cleanup test-latest-version; then
echo "cleanup hid the failed LINSTOR reclamation barrier" >&2
return 1
fi
}
@test "local Kubernetes suite cannot start without a LINSTOR baseline" {
# shellcheck source=/dev/null
. "$HACK_DIR/e2e-chainsaw/_lib/run-kubernetes.sh"
COZY_E2E_STORAGE_CLASS=local
yq() { printf 'v1.33.12\n'; }
kubectl() { return 0; }
cozy_wait_tenant_drained() { return 0; }
cozy_capture_linstor_pool_baseline() { return 42; }
helm() { printf 'UNEXPECTED_HELM_CALL\n'; return 99; }
rc=0
output=$(run_kubernetes_test '.' test-latest-version 59991) || rc=$?
if [ "$rc" -eq 0 ]; then
echo "suite accepted a failed local LINSTOR baseline capture" >&2
return 1
fi
if printf '%s\n' "$output" | grep -Fq UNEXPECTED_HELM_CALL; then
echo "suite continued into Helm after the baseline capture failed" >&2
return 1
fi
}
@test "Kubernetes cleanup operations outlive their bounded reclamation stages" {
for suite in latest previous; do
timeout=$(yq "select(.metadata.name == \"kubernetes-${suite}\") | .spec.steps[0].finally[0].script.timeout" "hack/e2e-chainsaw/kubernetes-${suite}/chainsaw-test.yaml")
if [ "$timeout" != 22m ]; then
echo "kubernetes-${suite} cleanup timeout is $timeout, expected 22m" >&2
return 1
fi
done
}
@test "both merge-gating lanes forward local storage through install and Chainsaw" {
# Checked in both workflows rather than one: pull-requests.yaml gates same-repo
# PRs and e2e-fork.yaml gates fork PRs, they carry the override separately, and
# a lane that installs on `local` while templating Chainsaw with `replicated`
# fails deep inside a suite rather than at the mismatch.
install_command=$(make -n -C packages/core/testing SANDBOX_NAME=test COZY_E2E_STORAGE_CLASS=local install-cozystack)
make_command=$(make -n -C packages/core/testing SANDBOX_NAME=test COZY_E2E_STORAGE_CLASS=local CHAINSAW_SUITES=vminstance test-chainsaw)
for wf in .github/workflows/pull-requests.yaml .github/workflows/e2e-fork.yaml; do
install_value=$(yq '.jobs.e2e.steps[] | select(.name == "Install Cozystack into sandbox") | .env.COZY_E2E_STORAGE_CLASS' "$wf")
run_value=$(yq '.jobs.e2e.steps[] | select(.name == "Run E2E tests") | .env.COZY_E2E_STORAGE_CLASS' "$wf")
if [ "$install_value" != local ] || [ "$run_value" != local ]; then
echo "unexpected storage modes in $wf: install=$install_value chainsaw=$run_value" >&2
return 1
fi
done
if ! printf '%s\n' "$install_command" | grep -Fq -- '-e COZY_E2E_STORAGE_CLASS="local"'; then
echo "testing Makefile does not forward COZY_E2E_STORAGE_CLASS into installation" >&2
return 1
fi
if ! printf '%s\n' "$make_command" | grep -Fq -- '-e COZY_E2E_STORAGE_CLASS="local"'; then
echo "testing Makefile does not forward COZY_E2E_STORAGE_CLASS into the sandbox" >&2
return 1
fi
if ! printf '%s\n' "$make_command" | grep -Fq -- '--set-string storageClass="${COZY_E2E_STORAGE_CLASS:-replicated}" vminstance'; then
echo "testing Makefile does not pass the lane storage class to Chainsaw values" >&2
return 1
fi
}
@test "container install supplies VictoriaLogs local storage before monitoring exists" {
install_suite=hack/e2e-install-cozystack.bats
if ! grep -Fq 'storage_class="${COZY_E2E_STORAGE_CLASS:-replicated}"' "$install_suite"; then
echo "install suite does not retain the replicated QEMU default" >&2
return 1
fi
if ! grep -Fq 'kubectl patch secret cozystack-values -n tenant-root --type merge --patch-file "$root_values_patch_file"' "$install_suite"; then
echo "install suite does not put the monitoring override in root values" >&2
return 1
fi
if grep -Fq 'kubectl patch hr/monitoring' "$install_suite"; then
echo "install suite still races monitoring reconciliation with a HelmRelease patch" >&2
return 1
fi
if ! grep -Fq '.logsStorages = [{"name":"generic","retentionPeriod":"1","storage":"10Gi","storageClassName":strenv(COZY_E2E_STORAGE_CLASS)}]' "$install_suite"; then
echo "monitoring override does not preserve the required VictoriaLogs values" >&2
return 1
fi
secret_patch_line=$(grep -nF 'kubectl patch secret cozystack-values' "$install_suite" | cut -d: -f1)
tenant_patch_line=$(grep -nF 'kubectl patch tenants/root' "$install_suite" | cut -d: -f1)
if [ -z "$secret_patch_line" ] || [ -z "$tenant_patch_line" ] || [ "$secret_patch_line" -ge "$tenant_patch_line" ]; then
echo "root values must be patched before monitoring is enabled: secret=$secret_patch_line tenant=$tenant_patch_line" >&2
return 1
fi
rendered=$(mktemp)
helm template monitoring packages/extra/monitoring --namespace tenant-root \
--set-string 'logsStorages[0].name=generic' \
--set-string 'logsStorages[0].retentionPeriod=1' \
--set-string 'logsStorages[0].storage=10Gi' \
--set-string 'logsStorages[0].storageClassName=local' > "$rendered"
rendered_class=$(yq 'select(.kind == "HelmRelease" and .metadata.name == "monitoring-system") | .spec.values.logsStorages[0].storageClassName' "$rendered")
rm -f "$rendered"
if [ "$rendered_class" != local ]; then
echo "monitoring chart did not carry the root values override into its child release: $rendered_class" >&2
return 1
fi
}
@test "DRBD-dependent fixtures and Kubernetes checks consume the lane storage mode" {
for manifest in hack/e2e-chainsaw/vminstance/vmdisk.yaml hack/e2e-chainsaw/vminstance/vmdisk-vmi.yaml; do
storage_class=$(yq '.spec.storageClass' "$manifest")
if [ "$storage_class" != '($values.storageClass)' ]; then
echo "$manifest does not consume the Chainsaw storageClass value: $storage_class" >&2
return 1
fi
done
kubernetes_script=hack/e2e-chainsaw/_lib/run-kubernetes.sh
rendered_uses=$(grep -Fc 'storageClass: "${storage_class}"' "$kubernetes_script")
if [ "$rendered_uses" -ne 2 ]; then
echo "expected both tenant Kubernetes resources to consume storage_class, found $rendered_uses" >&2
return 1
fi
if ! grep -Fq 'if [ "$storage_class" = local ]; then' "$kubernetes_script"; then
echo "local-only Kubernetes suites do not omit DRBD/RWX assertions" >&2
return 1
fi
if ! grep -Fq 'if [ "$storage_class" = replicated ]; then' "$kubernetes_script"; then
echo "fallback regression does not limit replicated StorageClass restoration to QEMU" >&2
return 1
fi
}