The in-sandbox Talos image mirror (#3244) is unreachable by the tenant
workers it serves, so the kubernetes-* e2e tests fail deterministically at
the 12m node-join wait instead of relieving the public-factory flake (#3231).
Root cause (from #3254 by @lexfrei, verified against the failing run's
crust-gather snapshot): tenant namespaces run a default-deny Cilium egress
(allow-internal/external-communication + the <tenant>-egress CCNP) that
permits the world entity and the tenant tree but NOT an arbitrary kube-system
Service, so a worker's CDI importer resolves talos-image-cache.kube-system.svc
yet its TCP connect to the ClusterIP is silently dropped and no disk imports.
#3244's gate never caught this because it probed the server over localhost
inside the serve pod, which passes regardless of cross-pod reachability.
Port of #3254 to the Chainsaw layout (hack/e2e-chainsaw/_lib/):
- ship a tightly-scoped CiliumClusterwideNetworkPolicy (importer pods in
tenant-test -> mirror pod in kube-system), applied at point-of-use once
Cilium CRDs exist; excluded from the pre-Cilium install apply via yq.
- replace the readiness probe with a throwaway pod in tenant-test labelled
cdi.kubevirt.io=importer that curls the ClusterIP with a range request and
must get 206 - the exact egress path a real importer faces; tenants use the
mirror only when that end-to-end check passes, else fall back to the public
factory, so it can only help, never make CI worse.
- unit test pinning the manifest split, probe/policy label agreement, the
strict 206 gate, and the --overrides JSON builder.
Also adds (a2) DataVolume import-stage diagnostics to the node-join failure
block in run-kubernetes.sh (importer logs + a ClusterIP re-probe via
talos_image_cache_diagnose) so the import-stage failure mode is legible.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The talos-image-cache serve container ran `busybox httpd`, but the pinned alpine/k8s image ships no `httpd` applet and no standalone httpd, so the container exited 127 and crash-looped and the Deployment never became Available. The resolver then burned its full 12-minute rollout wait before falling back to the public factory — a fixed latency tax on the first kubernetes-* test while the flake the mirror targets stayed live.
Serve the seeded image with a small range-capable server on the image's bundled python3 instead (stock `python3 -m http.server` ignores Range headers and answers 200, not 206). It needs no extra package pull, so the mirror comes up deterministically and keeps CDI's fast range import. The seed curl now also aborts and retries a stalled transfer (--speed-time/--speed-limit), and the resolver verifies a 206 range response before pointing tenants at the mirror.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The kubernetes-* e2e tests fail intermittently and run 18-22 min each
because every tenant worker VM imports its Talos OS disk directly from the
public factory.talos.dev: that endpoint has no byte-range support (forcing
CDI into a full copy-to-scratch) and its egress stream-resets / stalls
mid-transfer from the CI runner, so a worker DataVolume import hangs past
the chart's 12-minute node-join deadline and the test fails with zero
tenant nodes joined (observed on run 28924455578, and on the unrelated
branch in run 28929095889 - i.e. not specific to any one PR).
Deploy a best-effort in-sandbox mirror (hack/e2e-talos-image-cache.yaml)
during install: an initContainer fetches the image once with an aggressive
curl retry loop, then busybox httpd serves it locally with range support.
Tenant Kubernetes CRs point spec.talos.imageFactoryURL at its Service via
a shared helper (hack/e2e-apps/talos-image-cache.sh) that gates on the
mirror being Available and falls back to the public factory otherwise, so
the mirror can only help, never make CI worse. Deployed early so the seed
overlaps the Cozystack install; readiness is resolved once and cached
across bats files. Uses the overridable chart value added in the previous
commit.
Refs #3231
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
