Files
cozystack/hack/run-kubernetes-oidc_test.bats
T
Myasnikov Daniil 186547d835 test(e2e): run the merge-gating lanes on Talos containers
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.

The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.

Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.

Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.

The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.

Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.

One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.

Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-05 13:19:17 +05:00

125 lines
5.3 KiB
Bash

#!/usr/bin/env bats
# Unit coverage for the OIDC lifecycle folded into kubernetes-latest.
# Sourcing run-kubernetes.sh only defines functions; every kubectl interaction
# below is replaced with a shell stub, so this file needs no cluster.
@test "HelmRelease upgrade wait rejects stale Ready and stale observedGeneration" {
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
state_file=$(mktemp)
printf '0\n' > "$state_file"
kubectl() {
count=$(sed -n '1p' "$state_file")
count=$(( count + 1 ))
printf '%s\n' "$count" > "$state_file"
case "$count" in
1) printf '7|7|True|' ;;
2) printf '8|7|True|' ;;
*) printf '8|8|True|' ;;
esac
}
sleep() { :; }
cozy_wait_helmrelease_upgrade tenant-test kubernetes-demo 7 5 >/dev/null
[ "$(sed -n '1p' "$state_file")" -eq 3 ] || {
echo "the wait accepted Ready before both generation gates advanced" >&2
exit 1
}
}
@test "HelmRelease upgrade wait fails immediately on Stalled" {
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
calls=$(mktemp)
kubectl() {
printf '%s\n' "$*" >> "$calls"
case "$*" in
*" get helmrelease "*) printf '8|8|False|True' ;;
*" describe helmrelease "*) return 0 ;;
*) return 1 ;;
esac
}
sleep() { echo "sleep must not run after Stalled=True" >&2; return 1; }
if cozy_wait_helmrelease_upgrade tenant-test kubernetes-demo 7 600 >/dev/null 2>&1; then
echo "a Stalled HelmRelease was accepted" >&2
exit 1
fi
[ "$(wc -l < "$calls")" -eq 2 ] || {
echo "the Stalled path did not stop after status plus diagnostics" >&2
cat "$calls" >&2
exit 1
}
}
@test "System OIDC assertion proves rendered objects bootstrap kubeconfig and tenant RBAC" {
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
kubectl() {
case "$*" in
*" wait job kubernetes-demo-oidc-bootstrap "*) return 0 ;;
*" get kamajicontrolplane kubernetes-demo "*)
printf '[--authentication-config=/etc/kubernetes/authentication-config/config.yaml]'
;;
*" get secret kubernetes-demo-oidc-authn-config "*)
printf 'url: https://keycloak.example.test/realms/cozy\naudiences:\n- tenant-test-kubernetes-demo\n' | base64 | tr -d '\n'
;;
*" get keycloakclient.v1.edp.epam.com tenant-test-kubernetes-demo "*) printf 'true' ;;
*" get keycloakclientscope.v1.edp.epam.com tenant-test-kubernetes-demo-audience "*) printf 'oidc-audience-mapper' ;;
*" get secret kubernetes-demo-oidc-kubeconfig "*)
printf 'args:\n- oidc-login\n- --oidc-client-id=tenant-test-kubernetes-demo\n' | base64 | tr -d '\n'
;;
*) echo "unexpected kubectl call: $*" >&2; return 1 ;;
esac
}
cozy_oidc_bindings() {
printf 'e2e-admin@example.test\tcluster-admin\ne2e-viewer@example.test\tview\n'
}
cozy_assert_oidc_system demo
}
@test "CustomConfig transition patches the live cluster waits for the new generation and rejects System leftovers" {
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
calls=$(mktemp)
wait_args=$(mktemp)
patch_body=$(mktemp)
kubectl() {
printf '%s\n' "$*" >> "$calls"
case "$*" in
*" get helmrelease kubernetes-demo "*) printf '7' ;;
*" patch kuberneteses.apps.cozystack.io demo "*)
previous=
for argument in "$@"; do
if [ "$previous" = --patch ]; then
printf '%s\n' "$argument" > "$patch_body"
break
fi
previous="$argument"
done
;;
*" wait job kubernetes-demo-oidc-bootstrap "*) return 0 ;;
*" get kamajicontrolplane kubernetes-demo "*)
printf '[--authentication-config=/etc/kubernetes/authentication-config/config.yaml]'
;;
*" get secret kubernetes-demo-oidc-authn-config "*)
printf 'url: https://idp.byo.example.test\naudiences:\n- cozystack-byo-demo\n' | base64 | tr -d '\n'
;;
*" get keycloakclient.v1.edp.epam.com tenant-test-kubernetes-demo") return 1 ;;
*" get keycloakclientscope.v1.edp.epam.com tenant-test-kubernetes-demo-audience") return 1 ;;
*" get secret kubernetes-demo-oidc-kubeconfig") return 1 ;;
*) echo "unexpected kubectl call: $*" >&2; return 1 ;;
esac
}
cozy_wait_helmrelease_upgrade() { printf '%s\n' "$*" > "$wait_args"; }
cozy_oidc_bindings() { printf 'byo-admin@example.test\tcluster-admin\n'; }
cozy_switch_and_assert_oidc_custom_config demo
[ "$(sed -n '1p' "$wait_args")" = 'tenant-test kubernetes-demo 7 600' ] || {
echo "the transition did not wait from the pre-patch generation" >&2
cat "$wait_args" >&2
exit 1
}
jq -e '.spec.oidc.mode == "CustomConfig"' "$patch_body" >/dev/null
jq -e '.spec.oidc.users == [{"email":"byo-admin@example.test","role":"admin"}]' "$patch_body" >/dev/null
jq -er '.spec.oidc.customConfig.config' "$patch_body" | grep -q 'https://idp.byo.example.test'
jq -er '.spec.oidc.customConfig.config' "$patch_body" | grep -q 'cozystack-byo-demo'
}