The admission-webhook deny policies in ingress-nginx and kubeovn-webhook
cite Cilium source and the vendored values by line number. The bump
moved endpointLockdownOnMapOverflow and the policy-map ordering block in
pkg/endpoint/bpf.go, so the cited lines pointed elsewhere. Name the key
and the function instead, which survive the next bump, and move the
getEndpointSelector citation to v1.20.2, where the cited lines are
unchanged.
The kubeProxyReplacement pin explained the service translation in terms
of v1.19.5. With Gateway API on, v1.20.2 keeps socket LB in the host
namespace, so pod-namespace connections are translated per packet in
bpf_lxc before egress policy runs, which is what the pin relies on. The
same holds for the e2e image mirror, whose egress-policy comment in
hack/e2e-talos-image-cache.yaml described socket LB.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.
The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.
Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.
Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.
The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.
Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.
One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.
Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The in-sandbox Talos image mirror (#3244) is unreachable by the tenant
workers it serves, so the kubernetes-* e2e tests fail deterministically at
the 12m node-join wait instead of relieving the public-factory flake (#3231).
Root cause (from #3254 by @lexfrei, verified against the failing run's
crust-gather snapshot): tenant namespaces run a default-deny Cilium egress
(allow-internal/external-communication + the <tenant>-egress CCNP) that
permits the world entity and the tenant tree but NOT an arbitrary kube-system
Service, so a worker's CDI importer resolves talos-image-cache.kube-system.svc
yet its TCP connect to the ClusterIP is silently dropped and no disk imports.
#3244's gate never caught this because it probed the server over localhost
inside the serve pod, which passes regardless of cross-pod reachability.
Port of #3254 to the Chainsaw layout (hack/e2e-chainsaw/_lib/):
- ship a tightly-scoped CiliumClusterwideNetworkPolicy (importer pods in
tenant-test -> mirror pod in kube-system), applied at point-of-use once
Cilium CRDs exist; excluded from the pre-Cilium install apply via yq.
- replace the readiness probe with a throwaway pod in tenant-test labelled
cdi.kubevirt.io=importer that curls the ClusterIP with a range request and
must get 206 - the exact egress path a real importer faces; tenants use the
mirror only when that end-to-end check passes, else fall back to the public
factory, so it can only help, never make CI worse.
- unit test pinning the manifest split, probe/policy label agreement, the
strict 206 gate, and the --overrides JSON builder.
Also adds (a2) DataVolume import-stage diagnostics to the node-join failure
block in run-kubernetes.sh (importer logs + a ClusterIP re-probe via
talos_image_cache_diagnose) so the import-stage failure mode is legible.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The talos-image-cache serve container ran `busybox httpd`, but the pinned alpine/k8s image ships no `httpd` applet and no standalone httpd, so the container exited 127 and crash-looped and the Deployment never became Available. The resolver then burned its full 12-minute rollout wait before falling back to the public factory — a fixed latency tax on the first kubernetes-* test while the flake the mirror targets stayed live.
Serve the seeded image with a small range-capable server on the image's bundled python3 instead (stock `python3 -m http.server` ignores Range headers and answers 200, not 206). It needs no extra package pull, so the mirror comes up deterministically and keeps CDI's fast range import. The seed curl now also aborts and retries a stalled transfer (--speed-time/--speed-limit), and the resolver verifies a 206 range response before pointing tenants at the mirror.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The kubernetes-* e2e tests fail intermittently and run 18-22 min each
because every tenant worker VM imports its Talos OS disk directly from the
public factory.talos.dev: that endpoint has no byte-range support (forcing
CDI into a full copy-to-scratch) and its egress stream-resets / stalls
mid-transfer from the CI runner, so a worker DataVolume import hangs past
the chart's 12-minute node-join deadline and the test fails with zero
tenant nodes joined (observed on run 28924455578, and on the unrelated
branch in run 28929095889 - i.e. not specific to any one PR).
Deploy a best-effort in-sandbox mirror (hack/e2e-talos-image-cache.yaml)
during install: an initContainer fetches the image once with an aggressive
curl retry loop, then busybox httpd serves it locally with range support.
Tenant Kubernetes CRs point spec.talos.imageFactoryURL at its Service via
a shared helper (hack/e2e-apps/talos-image-cache.sh) that gates on the
mirror being Available and falls back to the public factory otherwise, so
the mirror can only help, never make CI worse. Deployed early so the seed
overlaps the Cozystack install; readiness is resolved once and cached
across bats files. Uses the overridable chart value added in the previous
commit.
Refs #3231
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>