Both new cases could report a verdict they had not observed, and the first CI
run of them did exactly that against a crash-looping ingress-nginx controller.
admission-webhooks-survive-namespace-churn died in setup: the admission Service
had no ready backend, so the first apply -- which passes through that same
fail-closed webhook -- failed with "no route to host" before any churn ran, and
the failure read as the outage under test. It now asserts both webhooks are up
BEFORE the churn, resolving their coordinates from the webhook registrations
themselves rather than from a guessed release name, and says so in its own
words when they are not. Proven against the pre-change script with a stubbed
kubectl: with a dead admission Service the old script proceeded into the churn
(delete namespace, two applies, delete namespace, two dry-runs); the new one
stops and names the cause without touching the churn namespace.
admission-webhook-deny-still-enforced measured a corpse. Its `kubectl wait` did
observe both pods Ready, but the controller was flapping on a ~100s cycle and
the probe ran against a dead backend 19 seconds later. "Denied by policy" and
"no backend to route to" are the same observation from inside the probe, so the
two deny assertions would have reported success against a cluster with no
policy installed at all, and the port-scoping assertion reported a policy bug
that did not exist. The deny verdicts are now gated on the admission Service
having ready backends before the probe, on every poll while it runs, and after
it finishes, and on the controller pods neither restarting nor being replaced
across that window -- EndpointSlice readiness rather than pod readiness,
because a `kubectl wait` is what a flapping pod satisfies transiently. The
probe reports reachable/unreachable per destination and always exits 0; this
script holds the endpoint evidence and is the only place that can attribute a
refused connect, so anything undecidable now fails as a setup failure instead
of as a verdict.
Ten scenarios driven through both scripts under dash with a stubbed kubectl,
all behaving as specified: a dead backend, a backend dying mid-probe, a
controller restarting mid-probe, a probe that never completed, an unregistered
webhook, the deny lost, the deny widened to every 443 destination, and the
healthy path. The restart-mid-probe case is the one that used to produce a
confident false pass.
Two more things the same rewrite fixes. The step budget was 6m against inner
budgets summing to 420s, so a slow-but-healthy cluster was killed with no
verdict and no describe output; both steps now carry a timeout above their own
sum, per the rule in hack/e2e-chainsaw/.chainsaw.yaml, and the `kubectl wait`
whose 180s dominated that sum is gone. The churn namespace now carries a
running pod, because a Cilium identity is a set of endpoint labels including
k8s:io.kubernetes.pod.namespace: an empty namespace allocates no identity and
its deletion drives none of the UpdateIdentities path the old target-side
policies coupled the webhook endpoint to. And both server-dry-run Pods carry a
restricted-compliant securityContext -- never scheduled, but a Pod that
PodSecurity would reject aborts the check under set -eu and reads as the
webhook being down.
Refs #3983.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Myasnikov Daniil
cbd0e78cad