Aleksei Sviridkin
ae5b626c8f
The chart README gains the native-port field, the three limitations it carries and where each comes from, the one that is a phase-1 narrowing rather than a consequence of the Cilium pin, and the three ways forward for anything that depended on a terminate listener the change takes away. The withdrawal is stated as what it is, a change of serving path rather than the removal of something idle, because on the pinned version both filter chains carry the SNI today and nothing on the objects says which one answers. The layers section is brought back in step with the listeners: which label each listener class selects on, that the attach set is the whole inheriting tenant subtree rather than the tenant plus the admin list, and the route conditions an operator reads, including the one that says the sectionName itself names nothing. The platform values comment on gateway.enabled said every exposed service renders an HTTPRoute, which the chart value the withdrawal reads contradicts. It now separates them: the services whose TLS the Gateway terminates get an HTTPRoute, while api, vm-exportproxy and cdi-uploadproxy get a TLSRoute pinned to their passthrough listener, one each from the packages that own those services. The overlap paragraph no longer describes an HTTPRoute wildcard as keeping its terminate listener. Such a claim is refused under HTTP-01, which issues no wildcard certificate, and the paragraph says so and what the render used to do with it. The route-condition list gains that shape and the port shapes an HTTPRoute now reaches, and the per-app naming bullets carry the hash suffix the rendered names have. Two sentences that were false against the code are corrected: a TLSRoute on a wildcard entry itself leaves the names beneath it their terminate listeners, and an unusable apex is refused by the containment rule at admission before the controller's own check is reached. The edge section says what happens to the entries the HTTP-01 pass wrote once the mode moves. The NoMatchingParent sentence now separates the two kinds: an HTTPRoute pinned to an absent passthrough section is refused whatever answers its hostname, while a TLSRoute is judged where a passthrough listener answers it. A sectionName outside the passthrough prefix is written Accepted like any attached route rather than left without a verdict, and the text says so; the port-80 listener's refusal joins the NotAllowedByListeners shapes and the name-and-port disagreement the NoMatchingParent ones. The attachedNamespaces comment described the list as reaching every listener, which the native-port ones do not: they select on kubernetes.io/metadata.name and name the publishing tenant alone, so a namespace listed there cannot attach to a database port. It also credited the wrong label with keeping the list honest; what a listed entry gets is the namespace.cozystack.io/gateway label, and what keeps a tenant from stamping it on itself is that no tenant role carries patch on Namespace. The class reference is brought along. Its certificate-mode table promised one Certificate per published hostname under http01, which a hostname served by a passthrough listener no longer earns, and its edge table said every attached TLSRoute is refused, where only a route declaring hostnames is and the rest have this controller's entry removed instead. The refusal-shape list is corrected against the code, without a count of its own so the next correction does not have to move one. A route the plain-HTTP listener turns away on its namespace selector was listed under NoMatchingListenerHostname as well as under NotAllowedByListeners; the controller reports the second. The port shape standing beside it swept port 80 back in by naming every port but 443, where a route selecting the plain-HTTP listener is answered by the namespace shapes and never reaches the port refusal at all. And a TLSRoute pinned to an absent section is said to carry NoMatchingListenerHostname where no passthrough listener answers its hostname, which holds only while no HTTPRoute claims that hostname: with one there a terminate listener stands on the name, and the route is judged like any other attached route. The wildcard shape gains the same qualifier. A wildcard claim is refused because no terminate listener can carry a certificate HTTP-01 will not issue, so the refusal reaches a route that wanted a terminate listener and not one that asked for the plain-HTTP listener, which carries no hostname and serves the claim there. The Helm merge paragraph named the two-way path as what happens to a custom resource Helm has no Go type for. That path is one of two in createPatch, the version deciding which runs, and the version comes from helm-controller rather than from this tree, so the claim was not checkable where it was written. It now names both and says the outcome it rests on holds either way. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
13 KiB
GatewayClass selection and TLS termination for tenant Gateways
A tenant that owns a Gateway (tenant.spec.gateway: true) gets one Gateway object bound to one GatewayClass. Which class that is decides who programs the dataplane and, with it, whether TLS is terminated by the Gateway itself or by something in front of it. This page covers the operator-facing surface: the platform values that bound the choice, the tenant field that makes it, what each resulting certificate mode implies, what a move between classes does to certificates and Secrets, and how the mechanism fails when the configured names and the installed classes disagree.
The chart-level reference for the per-tenant Gateway — inheritance, listener shapes, the security model, ACME rate limits — is packages/extra/gateway/README.md.
Platform values
Three keys under gateway in the platform values bound which class a tenant Gateway can land on:
gateway:
enabled: true
className: cilium
tenantSelectableClasses: []
edgeTerminatedClasses: []
className— the class every tenant Gateway uses unless the tenant names another one. Defaults to the Cilium class the platform ships.tenantSelectableClasses— the additional class names a tenant may name for itself. Empty (the default) means a tenant cannot name anything but the platform default.edgeTerminatedClasses— the class names whose provider terminates TLS upstream of the Gateway. Membership in this list is the only thing that puts a tenant into edge mode.
All three reach the per-tenant gateway chart through the platform values channel, as _cluster.gateway-class-name, _cluster.gateway-tenant-classes and _cluster.gateway-edge-classes. The two lists are joined with commas on the way, so a name containing a comma is rejected at platform render time rather than silently split into two names.
gateway.enabled is a separate switch and not a prerequisite for any of this: it moves the platform's own published services from Ingress onto Gateway API and points the ACME ClusterIssuers' HTTP-01 solver at the publishing tenant's Gateway. A tenant Gateway is created from tenant.spec.gateway alone.
How a tenant picks a class
apiVersion: apps.cozystack.io/v1alpha1
kind: Tenant
metadata:
name: acme
namespace: tenant-root
spec:
gateway: true
gatewayClass: edge-tunnel
edge-tunnel throughout this page stands for the name of a GatewayClass whose controller terminates TLS outside the cluster and forwards cleartext to the Gateway. Substitute the name your own edge-terminating controller registers.
The field is read only for a tenant that owns a Gateway. A tenant that inherits — gateway left unset or set to false — publishes through the nearest ancestor that owns one and runs on that ancestor's class, so its own gatewayClass has no effect. Leaving gatewayClass empty selects gateway.className.
The set a tenant may name is gateway.tenantSelectableClasses plus the current gateway.className, so a tenant may always name the default explicitly. Anything else fails that tenant's own gateway release at render time, with a message naming the class and the allowed set, and reaches no other tenant. The allowlist exists because Tenant is a tenant-writable object while a GatewayClass is cluster-scoped: without it a tenant could attach its Gateway to any class the cluster registers, including one an admin keeps for internal-only traffic.
What the class decides: the certificate mode
TenantGateway.spec.certMode is derived, never written by the tenant. The gateway chart resolves it in this order, first match winning:
| Condition | certMode |
What the controller issues |
|---|---|---|
the resolved class is in gateway.edgeTerminatedClasses |
edge |
nothing |
publishing.certificates.wildcardSecretName is set |
existingSecret |
nothing; the operator's Secret is referenced |
publishing.certificates.solver: dns01 |
dns01 |
one wildcard Certificate per tenant |
| otherwise | http01 |
one Certificate per published hostname, except the passthrough names below |
Under http01 a hostname held by a TLS-passthrough listener is the exception to the per-hostname rule. Once a TLSRoute is served on that listener and forwards to a backend, the hostname earns neither an HTTPS-terminate listener nor a Certificate, because the backend presents its own certificate and the Gateway never holds a key for the name. The shipped tlsPassthroughServices defaults put api, vm-exportproxy and cdi-uploadproxy in that position on the publishing tenant, and the Certificate objects that covered them are collected on upgrade; packages/extra/gateway/README.md carries what that changes and the ways forward.
Edge therefore wins over an operator-supplied wildcard Secret, which wins over the solver. In edge mode the solver, DNS-01 provider and issuer-name inputs are not read at all, so an unsupported value in any of them does not fail the tenant's release. existingSecret skips the same validations; only http01 and dns01 reject a bad solver, provider or issuer name.
What edge mode changes on the Gateway
- The Gateway serves port 80 only. Its listeners are
edgefor*.<apex>,edge-apexfor<apex>, and oneedge-child-*per inheriting child tenant carrying that child's*.<child-apex>. There is no port-443 listener, and no hostname-less port-80 listener either. - No per-tenant ACME
Issuerand noCertificateare created. - No http-to-https redirect
HTTPRouteis created. The edge is expected to perform that redirect. tlsPassthroughServicesis not written onto theTenantGatewayand no passthrough listener is rendered, so the tenant Kubernetes API, VM export and CDI upload endpoints are not published through such a Gateway. The packages that own those endpoints still create theirTLSRouteobjects, which then match no listener on that Gateway.- Application
HTTPRoutes attach by hostname exactly as in the other modes — the edge listeners carry the same namespace selector the HTTPS listeners carry elsewhere.
That last point has a blast radius worth stating on its own, and it applies only where those endpoints are published through the Gateway at all. Each of them renders its TLSRoute only when gateway.enabled is on and its name is still in publishing.exposedServices; at the default gateway.enabled: false there is no such route and nothing to lose. Where both hold, setting gateway.className to an edge-terminated class puts the publishing tenant on it too, and then the endpoints that go unpublished are the platform's own: the TLSRoute for api.<root-host> is pinned to a tls-api listener, an edge Gateway has no such listener, so the tenant Kubernetes API stops being served — VM export and CDI upload go the same way. Nothing fails at render time, and the loss surfaces only on the routes themselves, which the controller flips to Accepted=False with reason NoMatchingParent. Putting only child tenants on the edge class avoids it, because the platform's routes keep pointing at the publishing tenant's Gateway.
Listing a class in gateway.edgeTerminatedClasses is an assertion about that class, and nothing verifies it. Put the bundled cilium class in the list and its Gateways serve every application hostname over plain HTTP on a public address, with no redirect and no certificate. Only list a class whose provider really does terminate TLS in front of the Gateway.
Moving a tenant between classes
Onto an edge-terminated class, on the next reconcile:
| Object | What happens |
|---|---|
the per-tenant ACME Issuer |
deleted |
per-listener and wildcard Certificate objects |
deleted |
the TLS Secrets those Certificates issued |
left in the namespace, unreferenced |
the http-to-https redirect HTTPRoute |
deleted |
the replica of publishing.certificates.wildcardSecretName |
deleted from that tenant's namespace |
| route status conditions the controller wrote earlier | an attached TLSRoute that declares hostnames is flipped to Accepted=False, reason NoMatchingParent; on a hostname-less TLSRoute, and on every attached HTTPRoute, the controller's own entry is removed instead |
Every deletion is guarded by ownership: an object of the same name that the controller did not create is left in place.
The TLS Secrets survive because cert-manager ships here with enableCertificateOwnerRef: false, so a Certificate does not own the Secret it issued into. That is how every mode transition behaves and is not something edge introduces. Removing the leftovers is an operator action.
The wildcard replica is withdrawn because the namespace stops counting as a TLS termination point. Three things keep it anyway: a namespace that also owns an ingress controller still terminates locally; a namespace holding more than one TenantGateway keeps the replica while any of them is not on an edge-terminated class; and a namespace whose TenantGateway objects cannot be listed, or which carries the Gateway owner label but has no TenantGateway yet, keeps it as well, so that a transient read failure never withdraws a key a Gateway may still be serving.
Moving back off an edge-terminated class restores whatever the new mode calls for: the ACME Issuer and its Certificates under http01 or dns01, the redirect route under any non-edge mode, and the wildcard replica whenever publishing.certificates.wildcardSecretName is set. The TLSRoute conditions go back too, since the passthrough listeners do: http01 judges those routes again from the hostname claims it collects, while dns01 and existingSecret collect none and so drop the entry rather than leave a withdrawal standing on a listener that is serving. existingSecret mints no Issuer and no Certificate of its own, so a move from edge into it brings back the replica and the redirect route and nothing else.
The two halves of the move are not ordered against each other. The wildcard-secret controller watches the TenantGateway and prunes the replica as soon as certMode changes, while the Gateway itself is re-rendered when the tenant's HelmRelease reconciles, and the class controller reprograms after that. A tenant leaving existingSecret can therefore be briefly serving HTTPS listeners whose Secret has already gone.
When the configured names and the installed classes disagree
Nothing in this mechanism checks that a class name corresponds to a GatewayClass the cluster actually has. Run kubectl get gatewayclass before putting a name into any of the three values.
- A name in
classNameortenantSelectableClassesthat matches no installed class. The tenant's release renders and theGatewayobject is created, but no controller claims it, so it gets no address and no listener status. Nothing fails at render time; the symptom is theTenantGatewayholdingReady=Falsewith reasonGatewayNotAccepted. - A misspelled name in
edgeTerminatedClasses. It matches no class, so edge mode never engages for it and nothing says so. A tenant whose TLS really is terminated upstream stays in whichever mode the solver and wildcard-secret settings select, and its Gateway keeps terminating TLS itself. - A class removed from
tenantSelectableClasseswhile a tenant still names it. That tenant's gateway release starts failing to render. Other tenants are unaffected. classNamechanged while a tenant pins the outgoing name. The allowed set is the current default plustenantSelectableClasses, so the pin stops being valid and that tenant's gateway release fails. Add the outgoing class totenantSelectableClassesbefore changing the default, or update the tenant first.- The publishing tenant selecting a class that terminates TLS differently from
className. Refused at render time, in both directions. The cluster-wide ACMEClusterIssuers point their HTTP-01 solver at the publishing tenant's Gateway, and whether that solver pinssectionName: httpis decided fromgateway.classNamealone — a tenant's own choice does not travel on that channel. A publishing tenant that moved itself onto an edge class would keep a solver pinned to thehttplistener its Gateway no longer renders, and every HTTP-01 certificate in the cluster would stop issuing with nothing in any status naming the cause. Changegateway.classNameinstead. The publishing tenant is the namespace named bypublishing.ingressName,tenant-rootby default. - A comma or a non-string entry in either list. The platform chart fails to render, naming the entry.
Checking what a tenant landed on
kubectl get gatewayclass
kubectl --namespace tenant-acme get tenantgateway cozystack \
--output jsonpath='{.spec.gatewayClassName}{"\t"}{.spec.certMode}{"\n"}'
kubectl --namespace tenant-acme get tenantgateway cozystack \
--output jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status}{"\t"}{.reason}{"\n"}{end}'
kubectl --namespace tenant-acme get gateway cozystack \
--output jsonpath='{range .spec.listeners[*]}{.name}{"\t"}{.port}{"\t"}{.hostname}{"\n"}{end}'
kubectl --namespace tenant-acme get certificate,issuer
A tenant on an edge-terminated class reports certMode: edge, shows port-80 listeners only, and has no Certificate or Issuer that this controller created. Leftover TLS Secrets from an earlier mode are not shown by the last command — list Secrets of type kubernetes.io/tls in the namespace to find those.