Commit Graph
4 Commits
Author SHA1 Message Date
Aleksei Sviridkin ad3f15d4f3 feat(gateway)!: reserve a passthrough listener's name by the spec alone
The TenantGateway controller withheld the HTTPS-terminate listener for a
name a passthrough listener answers only once a TLSRoute was attached
there and resolved a backend, mirroring how Cilium builds its filter
chains. That kept a served endpoint up on Cilium 1.19.5, but it tied
every route's status to one implementation's translator and needed
Service and ReferenceGrant watches to land the deferred decision.

The shipped operator accepts a wildcard terminate listener beside a
narrower passthrough listener on 443 and keeps conflicting every other
overlap. A terminate listener for the passthrough name itself is not
narrower, so on 443 the rule now protects the dataplane outright; on a
native port Gateway API would serve the pair, and the rule holds as a
policy that keeps a platform name with its passthrough listener. Reading
it off the spec makes the outcome the same on every implementation.

Under http01 an HTTPRoute claiming such a name gets no listener and no
certificate whether or not a TLSRoute is attached. A route whose every
hostname is reserved reads Accepted=False with
NoMatchingListenerHostname; one that also claims a free hostname stays
Accepted and is served there, as Gateway API accepts a route whose
hostnames intersect at least one listener. dns01 and existingSecret are
unchanged: their wildcard listener still covers such a name, and the
passthrough listener answers it by its exact server name.

The chart writes the passthrough services on the publishing tenant
alone, so the names reserved by default are the ones the platform
publishes there through its own TLSRoutes. The shape an upgrade
changes is an HTTPRoute on such a name whose platform TLSRoute forwards
nowhere, because it never attached (the package owning the service is
not installed) or because none of its backends resolve: that route kept
a terminate listener and now gets none.

The backend resolution, its watches and the RBAC rules that served them
go away.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 06:32:42 +03:00
Aleksei Sviridkin bdca165f4c feat(gateway): report a GatewayClass that does not serve TLSRoute
A GatewayClass lists the features it serves in status.supportedFeatures,
and TLS passthrough is TLSRoute. On a class that leaves it out the
passthrough listeners are accepted and serve nothing, so the
TenantGateway now reads Ready=False/GatewayClassUnsupported naming the
class instead of Ready. A class that lists no features at all is not
judged: implementations predating the field leave it empty.

The class writes that list after it is accepted and may change it when
its controller is upgraded, so a GatewayClass event requeues the
TenantGateways on it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 17:53:25 +03:00
Aleksei Sviridkin 2bf9ae4a27 feat(gateway)!: serve passthrough hostnames and stop terminating them
Each tlsPassthroughListeners entry renders a mode=Passthrough listener
on its own port, answering the entry's hostname by SNI and admitting
TLSRoutes from the publishing tenant's namespace alone. It declares no
allowedRoutes.kinds: at the pinned Cilium v1.19.5 a listener that
spells its kinds out has them applied to every route on the Gateway
rather than to itself, so naming TLSRoute here would reject every
HTTPRoute on the Gateway.

A hostname a passthrough listener answers no longer earns an
HTTPS-terminate listener or a Gateway-issued certificate. That is the
break, and it is keyed on service rather than on declaration: the same
Cilium builds a passthrough filter chain per route out of the backends
that survived resolution, so a listener no TLSRoute attaches to, and a
route whose backendRefs resolve to nothing, both match no ClientHello
and leave the terminate listener holding the name. Eligibility asks
what v1.19.5 asks and no more, the reference allowed and the Service
present, and fails closed on a read it cannot complete. Services and
ReferenceGrants are watched, so a withdrawal deferred on a missing
backend lands on the pass that sees it arrive, and the Service watch
takes only the creates and deletes, existence being all that is read.

Which hostname a route claims is answered the same way. spec.hostnames
is optional on a TLSRoute, and ComputeHosts substitutes the hostname of
each TLS listener the route's parentRefs select for a route that
declares none, so such a route carries the SNI and takes the name over
exactly as one that spells it out. A declared hostname broader than the
listener's is that substitution from the other side: the route is
served on the listener's name and not on the wildcard it wrote, so a
claim covering a reserved name counts as a claim on it.

The substitution reaches only the listeners that would admit the
route. A native-port listener takes the publishing tenant's namespace
alone, so Cilium never puts a route from elsewhere on it and never
serves that route on its hostname; borrowing the name anyway refuses
the route for a namespace, over a hostname its own object does not
carry, in the same pass that sheds a terminate listener because that
same route is serving a name it borrowed legitimately. A route that
names the listener in its sectionName keeps the borrow, since there
the refusal is the answer to what it asked for.

A route the Gateway renders no listener for hears it from this
controller instead of from Cilium alone: the hostname a passthrough
listener took, the hostname nothing answers, the sectionName or port
naming a listener that answers something else, the sectionName naming
no rendered passthrough listener, and the native-port listener that
admits the tenant alone, each with its own Gateway API reason and a
message naming the hostnames that lost. An HTTPRoute pinning a
rendered passthrough listener by sectionName is refused there too,
since such a listener forwards a stream it never terminates and takes
TLSRoute alone by protocol. Nothing else tells that route: the port-443
listeners name HTTPRoute in their kinds to keep every port-443 set
uniform, which the Gateway reports back as InvalidRouteKinds rather
than as a grant, and the pin reads a listener's kinds against every
route on the Gateway, so it reports the route accepted.

The field is confined to certMode http01, stated as the modes that can
carry it so one added later is refused until someone rules on it. The
wildcard modes cover the whole apex with one terminate listener the
pinned Cilium cannot keep apart from a passthrough listener beneath it,
and edge terminates TLS at the class provider and renders no TLS
listener at all.

An HTTPRoute claiming a wildcard is refused under http01 before the
race and the overlap are read. The HTTP-01 challenge issues no
wildcard certificate, so the terminate listener rendered for the claim
could never carry the name, and the listener name built from the
hostname's first label carried the asterisk, which the Gateway CRD
admits in a hostname and refuses in a name, so the apiserver refused
the whole Gateway write and every other listener on it froze where it
stood. The renderer now writes that label as "wildcard", so every
hostname the CRD admits yields a name it admits, and no name that was
already admissible moves.

That refusal is a statement about the terminate listener, so it is
judged where that listener is decided rather than ahead of everything
else. The port-80 listener carries no hostname and serves a wildcard
claim as readily as any other, so a route that selects it, by pinning
the port or by naming the listener, never needed the listener HTTP-01
cannot supply and is served rather than refused. Judged first, the
refusal reached that route too and reported it unserved while Cilium
was serving it in plaintext.

A parentRef port on an HTTPRoute is read the way servableOn reads it
for a TLSRoute. The terminate listener sits on 443, so a route pinning
another port attaches to no listener the render would give it, and it
used to earn a listener and an ACME certificate for a route that
attached to nothing. Port 80 is the plain-HTTP listener, which admits
the publishing tenant alone: a route from there pinning it is served
plain and earns neither a listener nor a refusal.

A route status write that fails is told apart by kind. A conflict or
a route that vanished between the list and the write is a race between
the two status writers this Gateway has: the reconcile requeues and
retries the write without pinning Ready=False on the tenant, and the
status pass runs after the http-to-https redirect, so such a race
cannot withhold the one object that guard exists to provide. A write
the apiserver will refuse on every pass still fails the reconcile,
after every other route has been written, because the condition is the
only object saying why a hostname is not served, and a pass that
reported success over a lost write left the route reading Accepted=True
until the resync.

The retryable half is logged on the way out. Declining to pin
Ready=False leaves the requeue as the only thing acting on it, and the
requeue says nothing: the tenant goes on reading Ready, no event is
recorded, and returning no error opts out of the backoff that would
widen the interval, so a write failing on every pass retries at a fixed
period indefinitely with nothing anywhere to show for it.

A TLSRoute claiming a hostname broader than the listener's is pulled
into the eligibility question for each name it covers, because the
pinned Cilium serves it on the listener's own name. That is right only
where the listener answering a name is named exactly that name. Under
a wildcard entry the substitution hands such a route the wildcard, so
its chain carries "*.db.<apex>" while the terminate chain carries the
exact name beneath, and Envoy matches an exact server name ahead of a
wildcard, so nothing collides; pulling the route in there withdrew a
served endpoint and handed its name to the database behind the
passthrough listener. The pull-in is now keyed on an exact listener
match.

Every condition this controller writes on an HTTPRoute, and the one it
writes on a TLSRoute declaring no hostnames, is written under http01,
and the whole-apex modes collect no claims, so a switch of certMode
left those verdicts standing over listeners the new mode does not
render. The whole-apex pass now drops this controller's entry from
HTTPRoutes and from hostname-less TLSRoutes; the edge refusal stays
confined to routes that declare hostnames, since a refusal written on
a lent hostname would have no path back.

Within http01 the same verdicts outlive their listeners for a second
reason, and no mode change is needed to reach it. The status pass
walks the routes that attached rather than the ones that claimed a
hostname, and retracts its entry from the difference, because a claim
set is not a route's own property once a hostname can be borrowed: a
TLSRoute declaring none takes the hostname of every passthrough
listener its parentRef selects, so removing that listener empties the
claim set while the route stands unchanged, and an HTTPRoute reaches
the same state by dropping its hostnames. Judged on claims alone such
a route keeps whatever the last pass that did judge it left behind,
describing a listener the Gateway no longer has.

An HTTPRoute pinned by sectionName to a passthrough listener this
Gateway does not render is refused with NoMatchingParent, the reading
a TLSRoute in the same shape already got, instead of earning a
terminate listener it could never attach to.

A route served on the port-80 listener, by parentRef port or by
naming it, earns no terminate listener and is refused nothing, but it
publishes the name from its namespace all the same, so it stays in the
hostname race. Every recount now ranks it with the field it recounts,
because a recount that left it out crowned the namespace that had lost
to it and dropped a loss the first count recorded: a third route
refused in a third namespace, or a passthrough entry declared for the
name, flipped the loser to Accepted=True.

Selecting the port-80 listener by name is judged as selecting it by
port: from a namespace the listener admits the route is served plain,
from anywhere else the listener turns it away and the route reads
NotAllowedByListeners, and a sectionName and a parentRef port that
select different listeners select none and read NoMatchingParent,
which is what Gateway API requires when both are given. Before this a
route naming the listener from a refused namespace earned a terminate
listener and a certificate it could never attach to.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-10 18:47:30 +03:00
Aleksei Sviridkin d2e851f63b feat: Gateway API support via Cilium
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.

The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.

publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.

Security model -- seven independent layers, all admission-time gates
fail-closed:

  1. Listener allowedRoutes namespace whitelist via the
     kube-apiserver-written kubernetes.io/metadata.name label;
     plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
     passthrough listeners restrict allowedRoutes.kinds to
     HTTPRoute / TLSRoute respectively.
  2. cozystack-gateway-hostname-policy VAP on Gateway.
  3. cozystack-gateway-attached-namespaces-policy VAP on Package --
     blocks tenant-* in gateway.attachedNamespaces.
  4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
     writes from non-trusted callers.
  5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
     writes to namespace.cozystack.io/host from non-trusted callers.
  6. Render-time fail in cozystack-basics for tenant-* in
     gateway.attachedNamespaces.
  7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
     scoped to tenant-* namespaces.

Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).

Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.

Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.

Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.

Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.

Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.

Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-05-25 17:07:02 +03:00