The TenantGateway controller withheld the HTTPS-terminate listener for a
name a passthrough listener answers only once a TLSRoute was attached
there and resolved a backend, mirroring how Cilium builds its filter
chains. That kept a served endpoint up on Cilium 1.19.5, but it tied
every route's status to one implementation's translator and needed
Service and ReferenceGrant watches to land the deferred decision.
The shipped operator accepts a wildcard terminate listener beside a
narrower passthrough listener on 443 and keeps conflicting every other
overlap. A terminate listener for the passthrough name itself is not
narrower, so on 443 the rule now protects the dataplane outright; on a
native port Gateway API would serve the pair, and the rule holds as a
policy that keeps a platform name with its passthrough listener. Reading
it off the spec makes the outcome the same on every implementation.
Under http01 an HTTPRoute claiming such a name gets no listener and no
certificate whether or not a TLSRoute is attached. A route whose every
hostname is reserved reads Accepted=False with
NoMatchingListenerHostname; one that also claims a free hostname stays
Accepted and is served there, as Gateway API accepts a route whose
hostnames intersect at least one listener. dns01 and existingSecret are
unchanged: their wildcard listener still covers such a name, and the
passthrough listener answers it by its exact server name.
The chart writes the passthrough services on the publishing tenant
alone, so the names reserved by default are the ones the platform
publishes there through its own TLSRoutes. The shape an upgrade
changes is an HTTPRoute on such a name whose platform TLSRoute forwards
nowhere, because it never attached (the package owning the service is
not installed) or because none of its backends resolve: that route kept
a terminate listener and now gets none.
The backend resolution, its watches and the RBAC rules that served them
go away.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Require a real admission ClusterIP and carry the EndpointSlice target UID into the direct-backend coordinate. Tie that coordinate to a non-empty current Pod snapshot so stale endpoints or masked list failures cannot produce a policy verdict.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Probe one exact ready admission EndpointSlice backend throughout the measurement window and reject missing or ambiguous Service ports. Ensure the kubeovn-webhook Helm suites run through package discovery.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The step's polls can wait 660s under a 6m op, and its cleanup trap
deletes two tenants with no bound on the wait. When the op ceiling
fires first, Chainsaw kills the shell: the poll that was running never
reports, the trap never runs, and both tenants stay behind for every
later suite.
The deletes keep waiting, child before parent, since a parent deleted
while its child is still uninstalling wedges its own cleanup. Each
delete runs under an outer timeout with a kill grace, because kubectl's
--timeout bounds only the wait after the request and not discovery or
a stalled request. The parent is deleted only once the child is gone.
A failed tenant delete fails the step with kubectl's error, its exit
status and the tenant left behind, rather than passing with the leak.
The op is sized above the polls plus those bounds, and a unit guard
holds it to that sum and the cleanup to that order.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Three phases on a tenant that owns its Gateway. The shipped passthrough
entries render with nothing routed to them and an HTTPRoute on
api.<apex> keeps its terminate listener, its certificate and
Accepted=True, which is the state a withdrawal keyed on the declaration
alone would take away. Attaching a TLSRoute to that listener, pointed
at a Service in the tenant's own namespace so it resolves, withdraws
all three and moves the route to NoMatchingListenerHostname. A third
phase patches in a native-port entry and reads the per-listener
readiness flag, which is the claim no test against a fake client can
settle.
The first two phases are one measurement rather than two: withdrawing
on demand is what dates the earlier assertion to settled state instead
of to a withdrawal that had not arrived yet.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The comment on the shared CEL string said what the hostname requirement
is but not what it is worth per kind. Two of the three schemas leave
spec.hostnames optional at every version the vendored bundle serves,
so for HTTPRoute and GRPCRoute this policy is the only rejection a
hostname-less route meets; tlsroutes requires the field at v1 and
v1alpha3 and not at v1alpha2, so there the policy closes the gap for
as long as v1alpha2 is served. State that where the requirement is
stated.
The comment on the port-80 allowedRoutes filter credited the
cert-manager namespace entry for letting ACME complete, which the
paragraph above it retracts: the solver route lives in the tenant
namespace, and that is what keeps ACME working. The same sentence
claimed the filter keeps app routes off port 80, and the Layer 1
paragraph of the gateway README claimed it too. It does not. A tenant
that owns its Gateway is its own gateway namespace, so its app routes
are admitted by this very filter; what it excludes is cozy-* and
inheriting children. Both copies now say that.
Nothing said why naming the apex on the redirect costs no coverage,
which is the question a reader has once the route stops matching every
host. It holds because of the other half of this change: the port-80
listener admits routes only from the tenant namespace and
cert-manager's, and the tightened route policy requires every route
there to declare hostnames inside the apex. An inheriting child does
not widen that set either, and the comment now says so at the point
the hostnames are chosen: the gateway hostname policy compares every
listener against the host label of the Gateway's own namespace, so a
child apex outside it never gets its listener, the Gateway write
carrying it being refused at admission whole, in reconcileGateway and
before this route is rendered, while a child apex under this one
passes that check and the wildcard already covers it.
The empty-apex guard in renderHTTPRedirect, and the test pinning it,
both read as though the guard is what spares an operator an unreadable
schema error. It is, but the condition stated for it named the number
of attached routes, which does not enter into it. In HTTP-01 the
listener set is built from route hostnames alone, so reconcileGateway
never reads the apex ahead of the guard however many routes attach.
DNS-01 and existingSecret render an apex listener, and any
tlsPassthroughServices entry builds <service>.<apex>, so in those the
operator meets the equivalent error against the Gateway first. State
that condition once, at the guard, and let the test defer to it rather
than carry a second copy that can drift; the test stays green either
way because the fake client does not validate against the CRD schema.
The e2e suite explained omitting the TLSRoute empty-list probe by the
v1 schema rejecting an empty list. Both halves of that are true and
the conclusion does not follow: the probe runs at v1alpha2, which
marks spec.hostnames neither required nor minItems, so an empty list
reaches admission there exactly as an absent field does. What makes
the omission safe is that all three policies render one byte-identical
CEL string, so the HTTPRoute pair proves that term already.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The route hostname policy short-circuited on an absent field. Its CEL
opened with `!has(object.spec.hostnames) ||`, so an HTTPRoute,
TLSRoute or GRPCRoute declaring no hostnames satisfied the first
operand and was
admitted without a single hostname compared against the tenant apex.
An explicitly empty list reached the same outcome by a different path,
because `all()` over an empty list is vacuously true.
A route that names no hostname does not match nothing. Under Gateway
API it inherits the hostnames of every listener it attaches to, so it
covers whatever that listener covers. Require the field to be present
and non-empty before validating it.
All three policies rendered by the template share one CEL string, so
HTTPRoute, TLSRoute and GRPCRoute pick the change up together. The denial message
now names the missing-hostnames case, and the Gateway e2e grep moves to
the part of that message common to every denial it issues.
The Gateway e2e suite carried the one fixture that relied on the old
behaviour: its parentRef probe declared no hostnames and is applied
into the suite's tenant namespace. It now reads the apex from the
namespace label and names a hostname under it, as the neighbouring
cases in that suite already did.
Radius is unchanged: the policy still matches only tenant-* namespaces.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Restoring the target-side pod deny put a cluster-spanning peer selector back on
the webhook endpoint, which is the exact thing the source-side rules were added
to take off it. The comments still said the coupling was "structural and
removed", and the release note still said deleting a namespace no longer
rewrites that endpoint's policy map. Both were true of the previous revision
and are not true of this one.
What ships now is two denies that fail in different directions: the source-side
one keeps enforcing while the webhook endpoint's policy is being recomputed or
has overflowed its map, the target-side one keeps enforcing where Cilium is not
the service load balancer, and neither subsumes the other. So the coupling is
not eliminated, it is made non-load-bearing -- whichever end is degraded the
other still denies, and the map writes are writes rather than an outage.
Eliminating it outright would mean dropping the target-side rule and accepting
a silent fail-open, which is the worse trade.
The chainsaw case's header claimed the deletion "briefly made fail-closed
admission webhooks unreachable". That is the unconfirmed hypothesis, and the
errno in the report contradicts it: EHOSTUNREACH cannot come from a policy
denial, which drops after connect() has returned. It now says what the case
actually asserts -- both webhooks still admit across a namespace deletion --
rather than claiming to reproduce a cause.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Three narrowing fixes to the two webhook cases.
The availability precondition resolved only the FIRST registration of each
webhook name (awk ... exit) while every matching registration fires on the
applies below, so an unrelated flaky ingress-nginx release would have aborted
an apply and been reported as a churn-induced outage. It now waits on all of
them and says which one it is waiting for. Split into require_service_up plus a
loop, indexed with sed rather than piped into `while read`, because a
pipeline's body runs in a subshell under dash and a `return` from it would not
reach the caller.
That split immediately produced its own bug, caught by the scenario harness:
POSIX functions share one scope, so the callee's `label="$1"` overwrote the
caller's copy and the second registration reported itself as "(1/2) (2/2)".
The callee now uses distinct names.
The positive control's Service was picked with `grep -v -- '-admission$' |
head -1`, which relies on ingress-nginx-controller sorting before
ingress-nginx-controller-metrics. Deterministic but undocumented, and the
control has to be the Service that actually exposes the https port, so
-metrics is now excluded explicitly.
Also drops a stale sentence: the churn pod's comment claimed the old
target-side policies coupled the webhook endpoint to identity churn, which
overstates a map write as an outage and is moot now that the target-side rule
is back. The pod is there because an empty namespace allocates no identity, so
without it the case would exercise nothing it names.
Eleven scenarios through both scripts under dash with a stubbed kubectl, all as
specified, the new multi-registration case included.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Both new cases could report a verdict they had not observed, and the first CI
run of them did exactly that against a crash-looping ingress-nginx controller.
admission-webhooks-survive-namespace-churn died in setup: the admission Service
had no ready backend, so the first apply -- which passes through that same
fail-closed webhook -- failed with "no route to host" before any churn ran, and
the failure read as the outage under test. It now asserts both webhooks are up
BEFORE the churn, resolving their coordinates from the webhook registrations
themselves rather than from a guessed release name, and says so in its own
words when they are not. Proven against the pre-change script with a stubbed
kubectl: with a dead admission Service the old script proceeded into the churn
(delete namespace, two applies, delete namespace, two dry-runs); the new one
stops and names the cause without touching the churn namespace.
admission-webhook-deny-still-enforced measured a corpse. Its `kubectl wait` did
observe both pods Ready, but the controller was flapping on a ~100s cycle and
the probe ran against a dead backend 19 seconds later. "Denied by policy" and
"no backend to route to" are the same observation from inside the probe, so the
two deny assertions would have reported success against a cluster with no
policy installed at all, and the port-scoping assertion reported a policy bug
that did not exist. The deny verdicts are now gated on the admission Service
having ready backends before the probe, on every poll while it runs, and after
it finishes, and on the controller pods neither restarting nor being replaced
across that window -- EndpointSlice readiness rather than pod readiness,
because a `kubectl wait` is what a flapping pod satisfies transiently. The
probe reports reachable/unreachable per destination and always exits 0; this
script holds the endpoint evidence and is the only place that can attribute a
refused connect, so anything undecidable now fails as a setup failure instead
of as a verdict.
Ten scenarios driven through both scripts under dash with a stubbed kubectl,
all behaving as specified: a dead backend, a backend dying mid-probe, a
controller restarting mid-probe, a probe that never completed, an unregistered
webhook, the deny lost, the deny widened to every 443 destination, and the
healthy path. The restart-mid-probe case is the one that used to produce a
confident false pass.
Two more things the same rewrite fixes. The step budget was 6m against inner
budgets summing to 420s, so a slow-but-healthy cluster was killed with no
verdict and no describe output; both steps now carry a timeout above their own
sum, per the rule in hack/e2e-chainsaw/.chainsaw.yaml, and the `kubectl wait`
whose 180s dominated that sum is gone. The churn namespace now carries a
running pod, because a Cilium identity is a set of endpoint labels including
k8s:io.kubernetes.pod.namespace: an empty namespace allocates no identity and
its deletion drives none of the UpdateIdentities path the old target-side
policies coupled the webhook endpoint to. And both server-dry-run Pods carry a
restricted-compliant securityContext -- never scheduled, but a Pod that
PodSecurity would reject aborts the check under set -eu and reads as the
webhook being down.
Refs #3983.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The churn case added alongside it proves the webhooks stay AVAILABLE. Nothing
proved they are still UNREACHABLE from pods, which is the property the
policies exist for, and no test anywhere checked that the deny is confined to
the webhook port -- which is why a rule that denied all TCP/443 egress
cluster-wide passed every existing assertion.
Probe from tenant-root, where the host cluster's ingress-nginx lives and where
allow-internal-communication permits pods to reach it, so the deny is the only
thing that can block the webhook port; from the isolated tenant-test the probe
would pass for the wrong reason. Three assertions against the same pods and
the same tenant egress allowances, differing only in port: the controller's
own HTTPS port must stay reachable, the admission Service ClusterIP must not
be, and the backend port must not be. The first is the one that separates a
scoped deny from a wildcard.
Uses busybox:musl, already staged on every node by hack/e2e-prepull-images.sh,
with a securityContext that satisfies the restricted PodSecurity profile.
Service names, ports and the backend port are discovered at runtime rather
than hardcoded, since the release name differs between installs.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Exercise ingress and pod admission immediately after deleting a tenant
namespace, and route ingress-nginx changes to the gateway suite.
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Ingress, Gateway, and Route hostname ValidatingAdmissionPolicies
compare tenant hostnames against the tenant apex with case-sensitive CEL.
The resource-side host is already lowercase — the apiserver's hostname
validation permits nothing else — but the apex operand
is the namespace.cozystack.io/host label value, which Kubernetes permits to
carry uppercase, and the Ingress policy additionally compares against the
templated root host, a free-form platform value that nothing validates for
case.
A mixed-case apex label wrongly denies a tenant its own legitimate
lowercase in-apex hostname. An uppercase root host is the dangerous
direction: the Ingress outside-root exclusion compares a lowercase host
against the uppercase literal, never matches, so the exclusion stops
excluding and the policy degenerates into admitting any non-wildcard host
— every non-wildcard platform-apex name a tenant asks for, including the
root apex itself. Only the anti-wildcard gate still holds.
Wrap both operands of every comparison in .lowerAscii() so the match
follows DNS case semantics — no behavior change for the all-lowercase
default. Each policy reads a label it does not own: the namespace host
label is settable out-of-band by a trusted caller, so none of them may
assume the apex arrives normalized.
The Gateway policy admits any listener hostname when the host label is
absent or empty, unlike its two siblings. That is the platform's one
documented fail-open exception, recorded in the threat model together with
its compensating control, so changing the posture is a security-model
decision rather than a side effect of normalizing case. This leaves the
behavior untouched and pins its current shape in the tests.
Covered by helm-unittest assertions pinning each comparison as a whole
operand pair, so dropping the normalization from any single operand is
caught; a rendering case that pins the uppercase-root-host escape shut; and
e2e probes that set a mixed-case apex on an isolated namespace and assert
both halves of the match — a lowercase in-apex HTTPRoute and Ingress are
still admitted, while a foreign apex and a hostname that ends with the apex
without a dot boundary are still denied.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- redis: assert both StatefulSet-templated PVCs Bound (replicas: 2
provisions two, only ordinal 0 was checked)
- gateway: || true each delete inside the three cleanup() traps — they
run under set -eu, so one transient delete failure skipped the
remaining cleanup commands
- bucket, backup(.disabled): guard the trap's kill against an unset
pf_pid (script can exit before the port-forward starts)
Not addressed on purpose: the sort(ports[*].port) suggestions on
clickhouse/kafka — sort() is among the JMESPath forms chainsaw v0.2.15
(still the latest release) mis-evaluates, which is why the suites use
the ordered (ports[*].port) projection; services keep spec order.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Main added cozystack-ingress-hostname-policy (d89ad7069) with 12 bats
cases after the Chainsaw port was cut; this appends them as one Test.
- VAP/binding installation asserted declaratively (binding must carry
Deny)
- rejects and accepts grouped by derivation needs: apex-derived rejects
(foreign-apex host, dot-boundary prefix-adjacent, foreign tls host,
wildcard matching the root apex), static catch-all rejects (hostless
rule, empty-string host, defaultBackend), own-apex accepts (in-apex
rule+tls, own-apex wildcard), and static external-domain accepts as a
declarative apply with auto-cleanup
- hostnames derived at runtime: the tenant apex from the test
namespace's namespace.cozystack.io/host label (suite convention) and
the platform ROOT apex from the same label on tenant-root, replacing
the bats' hardcoded test.example.org/example.org; evil${apex}
preserves the dot-boundary semantics of eviltest.example.org, and the
external-domain probes use the RFC 2606 .invalid TLD so they can
never collide with a real cluster apex
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.
- Declarative ports assert on status/conditions and concrete fields; the
repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
collapses into a single polling assert with structured failure diffs and
automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
init/unseal, kuberture split-horizon probes, vminstance, the gateway
admission/impersonation cases, and kubernetes-latest/previous, which wrap
the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
runtime, so they are host-independent.
CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.
Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Aleksei Sviridkin