From a92d7f17a0bd5780f0507eeb2bc0fd2eb038265c Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Sun, 23 Aug 2026 16:57:28 +0500 Subject: [PATCH] ci(e2e): record SVM feature exposure on the runner Adds one step to the temporary topology job, ahead of the existing ones, and changes nothing else on this branch. Why. Across 32 archived nightly runs plus recent PR runs, every green run carries "kvm_amd: Virtual GIF supported" in the runner's dmesg and no red run does: 62 kubernetes suite instances without the line, zero passes, against 11 of 12 passing with it. vGIF lets a guest execute CLGI/STGI without trapping to the hypervisor, and a guest only executes those when it is itself running a hypervisor. In this stack that is exactly the sandbox Talos nodes hosting the KubeVirt tenant VMs and nothing else, which is why the sandbox nodes boot normally every single time while the tenant workers crawl. Without vGIF every tenant-VM world switch costs the sandbox node two extra full VM exits, which burns CPU without advancing the guest -- invisible to every resource counter, and consistent with a runner that measures 54-73% idle with ~0.01% steal while its nested guests freeze for tens of seconds. The line is missing for one of two reasons with completely different owners, and this step separates them: parameters/vgif = Y with no dmesg line -> the CPUID bit was never exposed to this VM by the host hypervisor, so only the cloud provider can change it parameters/vgif = N -> the runner image turned it off, which is a one-line fix CPUID 0x8000000A is read directly rather than inferred from whether the kernel chose to print a line, and the neighbouring V_VMSAVE_VMLOAD and VNMI bits come with it since they are the same class of "let the guest do it in hardware" feature. The mitigation strings are captured for the same reason: they change the cost of a VM exit without moving any resource counter. Every group ends in one greppable VERDICT line so re-running this job samples the runner pool without parsing the output. Re-runs are the point: vGIF exposure varies between hosts of the same shape and CPU model, so the distribution is what matters, not a single reading. This also closes an instrumentation gap that cost the investigation a month. runner-identity.txt, the SOFT-RED node-join marker and the KVM exit counters were all added around 19-21 August, after every green run had already happened, so none of them can be compared across the green/red boundary. The only reason vGIF was visible at all is that the kvm_amd lines happen to fall inside the 200-line dmesg tail the report keeps. Assisted-By: Claude Signed-off-by: Myasnikov Daniil --- .../workflows/tmp-runner-cpu-topology.yaml | 86 +++++++++++++++++++ 1 file changed, 86 insertions(+) diff --git a/.github/workflows/tmp-runner-cpu-topology.yaml b/.github/workflows/tmp-runner-cpu-topology.yaml index 459ee3a84..1e4916ecc 100644 --- a/.github/workflows/tmp-runner-cpu-topology.yaml +++ b/.github/workflows/tmp-runner-cpu-topology.yaml @@ -28,6 +28,92 @@ jobs: steps: # No checkout: nothing here reads the repository. Every command is # guarded so a missing tool reports instead of failing the run. + # Added for the vGIF investigation. Every green e2e run carries + # "kvm_amd: Virtual GIF supported" in the runner's dmesg and no red run + # does (62 kubernetes suite instances without it, zero passes). vGIF lets + # a guest run CLGI/STGI without trapping, which only matters when the + # guest is itself a hypervisor -- exactly the sandbox Talos nodes hosting + # the KubeVirt tenant VMs, and nothing else in the stack. That is why the + # sandbox nodes boot fine every time while the tenant workers crawl. + # + # This step separates the two reasons the line can be missing, which have + # completely different owners: + # parameters/vgif = Y and no dmesg line -> the CPUID bit was not + # exposed to this VM by the host hypervisor; only the cloud can fix it + # parameters/vgif = N -> the runner image disabled it; a one-line fix + # Nothing else in the tree records either, so a month of investigation + # could not tell these apart. + - name: SVM nested-virtualisation feature exposure + run: | + echo "::group::kvm_amd feature lines as the kernel reported them" + # dmesg may be restricted to root; sudo is passwordless on these runners + (sudo dmesg 2>/dev/null || dmesg 2>/dev/null || true) \ + | grep -E "kvm_amd|kvm-amd|kvm:" || echo "(no kvm lines in dmesg)" + echo "::endgroup::" + + echo "::group::kvm_amd module parameters" + for f in /sys/module/kvm_amd/parameters/*; do + [ -r "$f" ] || continue + printf '%s = %s\n' "$(basename "$f")" "$(cat "$f" 2>/dev/null)" + done + echo "::endgroup::" + + echo "::group::CPUID 0x8000000A (SVM revision and feature flags)" + # Read the SVM leaf directly, so "bit absent" is stated rather than + # inferred from the kernel's decision to print a line or not. + cat > /tmp/svm.c <<'CEOF' + #include + #include + int main(void) { + unsigned a=0,b=0,c=0,d=0; + if (!__get_cpuid(0x8000000A, &a,&b,&c,&d)) { printf("leaf unavailable\n"); return 0; } + printf("eax=0x%08x ebx=0x%08x ecx=0x%08x edx=0x%08x\n", a,b,c,d); + printf("svm_revision=%u nasid=%u\n", a & 0xff, b); + printf("NP(bit0)=%u VMCBClean(bit5)=%u PauseFilter(bit10)=%u\n", + !!(d&(1u<<0)), !!(d&(1u<<5)), !!(d&(1u<<10))); + printf("V_VMSAVE_VMLOAD(bit15)=%u VGIF(bit16)=%u VNMI(bit25)=%u\n", + !!(d&(1u<<15)), !!(d&(1u<<16)), !!(d&(1u<<25))); + return 0; + } + CEOF + sed -i 's/^ //' /tmp/svm.c + if command -v gcc >/dev/null 2>&1; then + gcc -O0 -o /tmp/svm /tmp/svm.c 2>/dev/null && /tmp/svm || echo "(compile failed)" + else + echo "(no gcc; rely on the module parameters and dmesg above)" + fi + echo "::endgroup::" + + echo "::group::machine identity" + grep -m1 "model name" /proc/cpuinfo || true + grep -m1 "microcode" /proc/cpuinfo || true + printf 'nproc=%s\n' "$(nproc)" + uname -r + for k in shape availabilityDomain faultDomain shapeConfig/ocpus; do + printf '%s: %s\n' "$k" \ + "$(curl -sS -m 5 -H 'Authorization: Bearer Oracle' \ + "http://169.254.169.254/opc/v2/instance/$k" 2>/dev/null || echo '?')" + done + echo "::endgroup::" + + echo "::group::speculative-execution mitigations" + # These change the cost of a VM exit without moving any resource + # counter, which is the same class of cause as vGIF. + for f in /sys/devices/system/cpu/vulnerabilities/*; do + [ -r "$f" ] || continue + printf '%s: %s\n' "$(basename "$f")" "$(cat "$f" 2>/dev/null)" + done + echo "::endgroup::" + + # One greppable line, so a sweep over many re-runs aggregates without + # parsing the groups above. + vgif_dmesg=no + (sudo dmesg 2>/dev/null || dmesg 2>/dev/null || true) \ + | grep -q "kvm_amd: Virtual GIF supported" && vgif_dmesg=yes + vgif_param="$(cat /sys/module/kvm_amd/parameters/vgif 2>/dev/null || echo '?')" + vgif_cpuid="$( { /tmp/svm 2>/dev/null || true; } | sed -n 's/.*VGIF(bit16)=\([01]\).*/\1/p')" + echo "VERDICT vgif_dmesg=${vgif_dmesg} vgif_param=${vgif_param} vgif_cpuid=${vgif_cpuid:-?} cpu=$(grep -m1 'model name' /proc/cpuinfo | sed 's/.*: //' | tr ' ' '_') kernel=$(uname -r)" + - name: Report host CPU run: | echo "::group::lscpu"