Files
cozystack/hack/talos-image-cache_test.bats
T
Myasnikov DaniilandClaude Opus 4.8 ff0d0d77a7 test(e2e): make the talos image cache reachable by tenant workers (#3254)
The in-sandbox Talos image mirror (#3244) is unreachable by the tenant
workers it serves, so the kubernetes-* e2e tests fail deterministically at
the 12m node-join wait instead of relieving the public-factory flake (#3231).

Root cause (from #3254 by @lexfrei, verified against the failing run's
crust-gather snapshot): tenant namespaces run a default-deny Cilium egress
(allow-internal/external-communication + the <tenant>-egress CCNP) that
permits the world entity and the tenant tree but NOT an arbitrary kube-system
Service, so a worker's CDI importer resolves talos-image-cache.kube-system.svc
yet its TCP connect to the ClusterIP is silently dropped and no disk imports.
#3244's gate never caught this because it probed the server over localhost
inside the serve pod, which passes regardless of cross-pod reachability.

Port of #3254 to the Chainsaw layout (hack/e2e-chainsaw/_lib/):
- ship a tightly-scoped CiliumClusterwideNetworkPolicy (importer pods in
  tenant-test -> mirror pod in kube-system), applied at point-of-use once
  Cilium CRDs exist; excluded from the pre-Cilium install apply via yq.
- replace the readiness probe with a throwaway pod in tenant-test labelled
  cdi.kubevirt.io=importer that curls the ClusterIP with a range request and
  must get 206 - the exact egress path a real importer faces; tenants use the
  mirror only when that end-to-end check passes, else fall back to the public
  factory, so it can only help, never make CI worse.
- unit test pinning the manifest split, probe/policy label agreement, the
  strict 206 gate, and the --overrides JSON builder.

Also adds (a2) DataVolume import-stage diagnostics to the node-join failure
block in run-kubernetes.sh (importer logs + a ClusterIP re-probe via
talos_image_cache_diagnose) so the import-stage failure mode is legible.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-09 18:21:07 +05:00

141 lines
7.9 KiB
Bash

#!/usr/bin/env bats
# -----------------------------------------------------------------------------
# Unit tests for the in-sandbox Talos image cache manifest split and the
# importer-reachability contract it relies on.
#
# hack/e2e-talos-image-cache.yaml bundles four documents but they are applied in
# two phases: hack/e2e-install-cozystack.bats applies everything EXCEPT the
# CiliumClusterwideNetworkPolicy (its CRD does not exist before Cozystack is
# installed), and hack/e2e-chainsaw/_lib/talos-image-cache.sh applies ONLY that
# policy later, once Cilium is up. If a future document is added to the manifest
# and silently dropped from the pre-Cilium apply, or the Cilium document leaks
# into it (and errors on the missing CRD), the mirror breaks and e2e falls back
# to the flaky public factory. These tests pin that split.
#
# They also pin the load-bearing reachability invariant: the throwaway probe pod
# is a faithful proxy for a real CDI importer only if it carries the exact label
# and namespace the egress policy selects. If either side drifts, the probe can
# pass while real importers stay blocked (a false positive that makes CI worse).
#
# Finally they cover the two pure fragments of the decision itself, sourced from
# hack/e2e-chainsaw/_lib/talos-image-cache.sh (sourcing it has no cluster side
# effects — it only sets parameter defaults and defines functions): the strict
# 206 gate that decides whether tenants are pointed at the mirror at all, and the
# --overrides JSON builder whose silent breakage would drop every probe to the
# public factory. The kubectl orchestration around them needs a live cluster and
# is covered by the e2e run, matching how the other hack/ helpers are tested.
#
# Root cause + fix are from cozystack/cozystack#3254 (@lexfrei); this is the port
# to the Chainsaw layout (hack/e2e-chainsaw/_lib/).
#
# cozytest.sh's awk parser recognizes only @test blocks and a bare `}` on its
# own line; there is no bats `run` or `$status`, and setup()/teardown() are not
# honored. Each test runs under `set -eu -x`; assertions are direct shell tests
# that exit non-zero on failure. mikefarah yq prints `---` between matched
# documents, so document streams are compared with those separators stripped.
#
# Run with: hack/cozytest.sh hack/talos-image-cache_test.bats
# -----------------------------------------------------------------------------
@test "manifest documents partition into pre-Cilium apply plus the Cilium policy" {
manifest=hack/e2e-talos-image-cache.yaml
total=$(yq '.kind' "$manifest" | grep -vc '^---$')
excluded=$(yq 'select(.kind != "CiliumClusterwideNetworkPolicy") | .kind' "$manifest" | grep -vc '^---$')
selected=$(yq 'select(.kind == "CiliumClusterwideNetworkPolicy") | .kind' "$manifest" | grep -vc '^---$')
[ "$total" -eq 4 ]
[ "$excluded" -eq 3 ]
[ "$selected" -eq 1 ]
[ $((excluded + selected)) -eq "$total" ]
}
@test "pre-Cilium apply keeps Service, Deployment, ConfigMap and drops the Cilium policy" {
manifest=hack/e2e-talos-image-cache.yaml
kinds=$(yq 'select(.kind != "CiliumClusterwideNetworkPolicy") | .kind' "$manifest" | grep -v '^---$')
for want in Service Deployment ConfigMap; do
printf '%s\n' "$kinds" | grep -qx "$want" || { echo "pre-Cilium apply is missing $want" >&2; exit 1; }
done
if printf '%s\n' "$kinds" | grep -qx CiliumClusterwideNetworkPolicy; then
echo "CiliumClusterwideNetworkPolicy leaked into the pre-Cilium apply" >&2
exit 1
fi
}
@test "point-of-use apply selects exactly the Cilium policy" {
manifest=hack/e2e-talos-image-cache.yaml
kinds=$(yq 'select(.kind == "CiliumClusterwideNetworkPolicy") | .kind' "$manifest" | grep -v '^---$')
[ "$kinds" = "CiliumClusterwideNetworkPolicy" ]
}
@test "egress policy selects the importer label and namespace the probe pod uses" {
manifest=hack/e2e-talos-image-cache.yaml
helper=hack/e2e-chainsaw/_lib/talos-image-cache.sh
label=$(yq 'select(.kind == "CiliumClusterwideNetworkPolicy") | .spec.endpointSelector.matchLabels["k8s:cdi.kubevirt.io"]' "$manifest")
ns=$(yq 'select(.kind == "CiliumClusterwideNetworkPolicy") | .spec.endpointSelector.matchLabels["k8s:io.kubernetes.pod.namespace"]' "$manifest")
[ "$label" = "importer" ]
[ "$ns" = "tenant-test" ]
# The probe pod must carry the same label the policy selects, else it faces a
# different egress than a real importer and the guarantee breaks.
grep -q "cdi.kubevirt.io=importer" "$helper" || { echo "probe pod label drifted from the egress selector" >&2; exit 1; }
grep -q "TALOS_IMAGE_CACHE_PROBE_NS:-tenant-test" "$helper" || { echo "probe namespace drifted from the egress selector" >&2; exit 1; }
}
@test "egress rule targets the mirror pod's own identity" {
manifest=hack/e2e-talos-image-cache.yaml
target=$(yq 'select(.kind == "CiliumClusterwideNetworkPolicy") | .spec.egress[0].toEndpoints[0].matchLabels["k8s:app.kubernetes.io/name"]' "$manifest")
target_ns=$(yq 'select(.kind == "CiliumClusterwideNetworkPolicy") | .spec.egress[0].toEndpoints[0].matchLabels["k8s:io.kubernetes.pod.namespace"]' "$manifest")
mirror=$(yq 'select(.kind == "Deployment") | .spec.template.metadata.labels["app.kubernetes.io/name"]' "$manifest")
mirror_ns=$(yq 'select(.kind == "Deployment") | .metadata.namespace' "$manifest")
[ "$target" = "talos-image-cache" ]
[ "$target_ns" = "kube-system" ]
# The allow's destination must equal the mirror Deployment's own pod label and
# namespace, otherwise the hole is punched toward nothing: moving the mirror
# would leave these tests green while importers could no longer reach it.
[ "$target" = "$mirror" ]
[ "$target_ns" = "$mirror_ns" ]
}
@test "strict 206 gate selects the mirror on a byte-range success" {
. hack/e2e-chainsaw/_lib/talos-image-cache.sh
if ! _talos_image_cache_probe_succeeded "code=206"; then
echo "expected a 206 probe result to select the mirror" >&2
exit 1
fi
}
@test "strict 206 gate finds the marker inside surrounding probe output" {
. hack/e2e-chainsaw/_lib/talos-image-cache.sh
out=$(printf 'pod/talos-image-cache-probe created\ncode=206\npod "talos-image-cache-probe" deleted\n')
if ! _talos_image_cache_probe_succeeded "$out"; then
echo "expected 206 to be found in multi-line attach output" >&2
exit 1
fi
}
@test "strict 206 gate falls back on range-less, unreachable, empty and unrelated output" {
. hack/e2e-chainsaw/_lib/talos-image-cache.sh
# A plain 200 means the mirror answered without range support; 000 means curl
# never connected. Both fall back to the public factory, as does output that
# carried no result marker at all (an attach stream lost with no logs to recover).
for bad in "code=200" "code=000" "code=404" "" "error: pods not found"; do
if _talos_image_cache_probe_succeeded "$bad"; then
echo "expected non-206 output to fall back to the public factory: [$bad]" >&2
exit 1
fi
done
}
@test "probe overrides builder emits valid JSON with the image, command and restricted PSA" {
. hack/e2e-chainsaw/_lib/talos-image-cache.sh
img=example.invalid/img:tag
cmd='curl -s -o /dev/null -w code=%{http_code} -r 0-0 http://talos-image-cache.kube-system.svc/x.raw.xz'
json=$(_talos_image_cache_probe_overrides "$img" "$cmd")
# Invalid JSON would make every probe fail and silently drop CI back to the
# public factory, so parse it rather than pattern-match it.
printf '%s' "$json" | yq -p=json -o=json '.' >/dev/null
[ "$(printf '%s' "$json" | yq -p=json '.spec.containers[0].name')" = "c" ]
[ "$(printf '%s' "$json" | yq -p=json '.spec.containers[0].image')" = "$img" ]
[ "$(printf '%s' "$json" | yq -p=json '.spec.containers[0].command[2]')" = "$cmd" ]
[ "$(printf '%s' "$json" | yq -p=json '.spec.securityContext.runAsNonRoot')" = "true" ]
[ "$(printf '%s' "$json" | yq -p=json '.spec.containers[0].securityContext.allowPrivilegeEscalation')" = "false" ]
}