mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 22:13:18 +09:00
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.
The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.
Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.
Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.
The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.
Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.
One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.
Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.
Adapted for release-1.6: this branch has no e2e-fork.yaml, no soft-red
gate, no ghcr.io mirror and no backup e2e, so those hunks are dropped
together with the tests for helpers that exist only on main; e2e-tag.yaml
stays on QEMU. The platform Package generator carries the API endpoint
and nothing else, because the linstor drbd.enabled and kubevirt-cdi
importerResources overrides need chart values this branch does not ship;
the container lane gets e2e-side substitutes for them in later commits.
run-kubernetes.sh takes the storage-class helper, the local-lane pool
baseline, the scoped drain and the OIDC fold, while cleanup keeps its
best-effort `|| true`. The OIDC suite references in select-install.sh and
the chainsaw README go with the suites here rather than in a later commit.
(cherry picked from commit 186547d835)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-by: LLM
149 lines
8.4 KiB
Bash
149 lines
8.4 KiB
Bash
#!/usr/bin/env bats
|
|
# -----------------------------------------------------------------------------
|
|
# Unit tests for the in-sandbox Talos image cache manifest split and the
|
|
# importer-reachability contract it relies on.
|
|
#
|
|
# hack/e2e-talos-image-cache.yaml bundles four documents but they are applied in
|
|
# two phases: hack/e2e-install-cozystack.bats applies everything EXCEPT the
|
|
# CiliumClusterwideNetworkPolicy (its CRD does not exist before Cozystack is
|
|
# installed), and hack/e2e-chainsaw/_lib/talos-image-cache.sh applies ONLY that
|
|
# policy later, once Cilium is up. If a future document is added to the manifest
|
|
# and silently dropped from the pre-Cilium apply, or the Cilium document leaks
|
|
# into it (and errors on the missing CRD), the mirror breaks and e2e falls back
|
|
# to the flaky public factory. These tests pin that split.
|
|
#
|
|
# They also pin the load-bearing reachability invariant: the throwaway probe pod
|
|
# is a faithful proxy for a real CDI importer only if it carries the exact label
|
|
# and namespace the egress policy selects. If either side drifts, the probe can
|
|
# pass while real importers stay blocked (a false positive that makes CI worse).
|
|
#
|
|
# Finally they cover the two pure fragments of the decision itself, sourced from
|
|
# hack/e2e-chainsaw/_lib/talos-image-cache.sh (sourcing it has no cluster side
|
|
# effects — it only sets parameter defaults and defines functions): the strict
|
|
# 206 gate that decides whether tenants are pointed at the mirror at all, and the
|
|
# --overrides JSON builder whose silent breakage would drop every probe to the
|
|
# public factory. The kubectl orchestration around them needs a live cluster and
|
|
# is covered by the e2e run, matching how the other hack/ helpers are tested.
|
|
#
|
|
# Root cause + fix are from cozystack/cozystack#3254 (@lexfrei); this is the port
|
|
# to the Chainsaw layout (hack/e2e-chainsaw/_lib/).
|
|
#
|
|
# cozytest.sh's awk parser recognizes only @test blocks and a bare `}` on its
|
|
# own line; there is no bats `run` or `$status`, and setup()/teardown() are not
|
|
# honored. Each test runs under `set -eu -x`; assertions are direct shell tests
|
|
# that exit non-zero on failure. mikefarah yq prints `---` between matched
|
|
# documents, so document streams are compared with those separators stripped.
|
|
#
|
|
# Run with: hack/cozytest.sh hack/talos-image-cache_test.bats
|
|
# -----------------------------------------------------------------------------
|
|
|
|
@test "manifest documents partition into pre-Cilium apply plus the Cilium policy" {
|
|
manifest=hack/e2e-talos-image-cache.yaml
|
|
total=$(yq '.kind' "$manifest" | grep -vc '^---$')
|
|
excluded=$(yq 'select(.kind != "CiliumClusterwideNetworkPolicy") | .kind' "$manifest" | grep -vc '^---$')
|
|
selected=$(yq 'select(.kind == "CiliumClusterwideNetworkPolicy") | .kind' "$manifest" | grep -vc '^---$')
|
|
[ "$total" -eq 4 ]
|
|
[ "$excluded" -eq 3 ]
|
|
[ "$selected" -eq 1 ]
|
|
[ $((excluded + selected)) -eq "$total" ]
|
|
}
|
|
|
|
@test "pre-Cilium apply keeps Service, Deployment, ConfigMap and drops the Cilium policy" {
|
|
manifest=hack/e2e-talos-image-cache.yaml
|
|
kinds=$(yq 'select(.kind != "CiliumClusterwideNetworkPolicy") | .kind' "$manifest" | grep -v '^---$')
|
|
for want in Service Deployment ConfigMap; do
|
|
printf '%s\n' "$kinds" | grep -qx "$want" || { echo "pre-Cilium apply is missing $want" >&2; exit 1; }
|
|
done
|
|
if printf '%s\n' "$kinds" | grep -qx CiliumClusterwideNetworkPolicy; then
|
|
echo "CiliumClusterwideNetworkPolicy leaked into the pre-Cilium apply" >&2
|
|
exit 1
|
|
fi
|
|
}
|
|
|
|
@test "point-of-use apply selects exactly the Cilium policy" {
|
|
manifest=hack/e2e-talos-image-cache.yaml
|
|
kinds=$(yq 'select(.kind == "CiliumClusterwideNetworkPolicy") | .kind' "$manifest" | grep -v '^---$')
|
|
[ "$kinds" = "CiliumClusterwideNetworkPolicy" ]
|
|
}
|
|
|
|
@test "range server records transfer outcome, bytes and duration" {
|
|
script=$(yq 'select(.kind == "ConfigMap") | .data."serve.py"' hack/e2e-talos-image-cache.yaml)
|
|
|
|
printf '%s\n' "$script" | python3 -c 'import sys; compile(sys.stdin.read(), "serve.py", "exec")'
|
|
printf '%s\n' "$script" | grep -Fq 'transfer-finished status=%s path=%s range=%s requested=%d sent=%d duration=%.3fs'
|
|
printf '%s\n' "$script" | grep -Fq 'time.monotonic() - started'
|
|
}
|
|
|
|
@test "egress policy selects the importer label and namespace the probe pod uses" {
|
|
manifest=hack/e2e-talos-image-cache.yaml
|
|
helper=hack/e2e-chainsaw/_lib/talos-image-cache.sh
|
|
label=$(yq 'select(.kind == "CiliumClusterwideNetworkPolicy") | .spec.endpointSelector.matchLabels["k8s:cdi.kubevirt.io"]' "$manifest")
|
|
ns=$(yq 'select(.kind == "CiliumClusterwideNetworkPolicy") | .spec.endpointSelector.matchLabels["k8s:io.kubernetes.pod.namespace"]' "$manifest")
|
|
[ "$label" = "importer" ]
|
|
[ "$ns" = "tenant-test" ]
|
|
# The probe pod must carry the same label the policy selects, else it faces a
|
|
# different egress than a real importer and the guarantee breaks.
|
|
grep -q "cdi.kubevirt.io=importer" "$helper" || { echo "probe pod label drifted from the egress selector" >&2; exit 1; }
|
|
grep -q "TALOS_IMAGE_CACHE_PROBE_NS:-tenant-test" "$helper" || { echo "probe namespace drifted from the egress selector" >&2; exit 1; }
|
|
}
|
|
|
|
@test "egress rule targets the mirror pod's own identity" {
|
|
manifest=hack/e2e-talos-image-cache.yaml
|
|
target=$(yq 'select(.kind == "CiliumClusterwideNetworkPolicy") | .spec.egress[0].toEndpoints[0].matchLabels["k8s:app.kubernetes.io/name"]' "$manifest")
|
|
target_ns=$(yq 'select(.kind == "CiliumClusterwideNetworkPolicy") | .spec.egress[0].toEndpoints[0].matchLabels["k8s:io.kubernetes.pod.namespace"]' "$manifest")
|
|
mirror=$(yq 'select(.kind == "Deployment") | .spec.template.metadata.labels["app.kubernetes.io/name"]' "$manifest")
|
|
mirror_ns=$(yq 'select(.kind == "Deployment") | .metadata.namespace' "$manifest")
|
|
[ "$target" = "talos-image-cache" ]
|
|
[ "$target_ns" = "kube-system" ]
|
|
# The allow's destination must equal the mirror Deployment's own pod label and
|
|
# namespace, otherwise the hole is punched toward nothing: moving the mirror
|
|
# would leave these tests green while importers could no longer reach it.
|
|
[ "$target" = "$mirror" ]
|
|
[ "$target_ns" = "$mirror_ns" ]
|
|
}
|
|
|
|
@test "strict 206 gate selects the mirror on a byte-range success" {
|
|
. hack/e2e-chainsaw/_lib/talos-image-cache.sh
|
|
if ! _talos_image_cache_probe_succeeded "code=206"; then
|
|
echo "expected a 206 probe result to select the mirror" >&2
|
|
exit 1
|
|
fi
|
|
}
|
|
|
|
@test "strict 206 gate finds the marker inside surrounding probe output" {
|
|
. hack/e2e-chainsaw/_lib/talos-image-cache.sh
|
|
out=$(printf 'pod/talos-image-cache-probe created\ncode=206\npod "talos-image-cache-probe" deleted\n')
|
|
if ! _talos_image_cache_probe_succeeded "$out"; then
|
|
echo "expected 206 to be found in multi-line attach output" >&2
|
|
exit 1
|
|
fi
|
|
}
|
|
|
|
@test "strict 206 gate falls back on range-less, unreachable, empty and unrelated output" {
|
|
. hack/e2e-chainsaw/_lib/talos-image-cache.sh
|
|
# A plain 200 means the mirror answered without range support; 000 means curl
|
|
# never connected. Both fall back to the public factory, as does output that
|
|
# carried no result marker at all (an attach stream lost with no logs to recover).
|
|
for bad in "code=200" "code=000" "code=404" "" "error: pods not found"; do
|
|
if _talos_image_cache_probe_succeeded "$bad"; then
|
|
echo "expected non-206 output to fall back to the public factory: [$bad]" >&2
|
|
exit 1
|
|
fi
|
|
done
|
|
}
|
|
|
|
@test "probe overrides builder emits valid JSON with the image, command and restricted PSA" {
|
|
. hack/e2e-chainsaw/_lib/talos-image-cache.sh
|
|
img=example.invalid/img:tag
|
|
cmd='curl -s -o /dev/null -w code=%{http_code} -r 0-0 http://talos-image-cache.kube-system.svc/x.raw.xz'
|
|
json=$(_talos_image_cache_probe_overrides "$img" "$cmd")
|
|
# Invalid JSON would make every probe fail and silently drop CI back to the
|
|
# public factory, so parse it rather than pattern-match it.
|
|
printf '%s' "$json" | yq -p=json -o=json '.' >/dev/null
|
|
[ "$(printf '%s' "$json" | yq -p=json '.spec.containers[0].name')" = "c" ]
|
|
[ "$(printf '%s' "$json" | yq -p=json '.spec.containers[0].image')" = "$img" ]
|
|
[ "$(printf '%s' "$json" | yq -p=json '.spec.containers[0].command[2]')" = "$cmd" ]
|
|
[ "$(printf '%s' "$json" | yq -p=json '.spec.securityContext.runAsNonRoot')" = "true" ]
|
|
[ "$(printf '%s' "$json" | yq -p=json '.spec.containers[0].securityContext.allowPrivilegeEscalation')" = "false" ]
|
|
}
|