The TenantGateway controller withheld the HTTPS-terminate listener for a
name a passthrough listener answers only once a TLSRoute was attached
there and resolved a backend, mirroring how Cilium builds its filter
chains. That kept a served endpoint up on Cilium 1.19.5, but it tied
every route's status to one implementation's translator and needed
Service and ReferenceGrant watches to land the deferred decision.
The shipped operator accepts a wildcard terminate listener beside a
narrower passthrough listener on 443 and keeps conflicting every other
overlap. A terminate listener for the passthrough name itself is not
narrower, so on 443 the rule now protects the dataplane outright; on a
native port Gateway API would serve the pair, and the rule holds as a
policy that keeps a platform name with its passthrough listener. Reading
it off the spec makes the outcome the same on every implementation.
Under http01 an HTTPRoute claiming such a name gets no listener and no
certificate whether or not a TLSRoute is attached. A route whose every
hostname is reserved reads Accepted=False with
NoMatchingListenerHostname; one that also claims a free hostname stays
Accepted and is served there, as Gateway API accepts a route whose
hostnames intersect at least one listener. dns01 and existingSecret are
unchanged: their wildcard listener still covers such a name, and the
passthrough listener answers it by its exact server name.
The chart writes the passthrough services on the publishing tenant
alone, so the names reserved by default are the ones the platform
publishes there through its own TLSRoutes. The shape an upgrade
changes is an HTTPRoute on such a name whose platform TLSRoute forwards
nowhere, because it never attached (the package owning the service is
not installed) or because none of its backends resolve: that route kept
a terminate listener and now gets none.
The backend resolution, its watches and the RBAC rules that served them
go away.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The refusal existed because Cilium 1.19.5 served every Gateway port
from one Envoy listener matched by SNI alone, so the "*.<apex>"
terminate listener of the wildcard modes and a passthrough listener
under the apex could reach each other's backend. The shipped Cilium
1.20.2 gives each port its own Envoy listener when SNI overlaps across
ports, and Gateway API treats listeners on different ports as
compatible, so the pair is served as declared. edge still refuses the
field: it renders no TLS listener for an entry to bind to.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
An empty selector matches every Pod, PVC, Service and DataVolume in the
namespace, so one such monitor goes non-operational on any import there.
No chart renders one, but the schema marked the field required without
requiring any label, and the API accepted it.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
WorkloadsReady=False covers different situations: a stopped VM has
no pod and its monitor reports it not operational on purpose, while a
disk whose DataVolume never populates is broken. A consumer that wants
to react to one and not the other had only the message text to go by.
The WorkloadMonitor status gains a Reason next to Message, set to
DataVolumeNotReady by the DataVolume check, and the Application API
carries it into the WorkloadsReady condition in place of the generic
reason whenever a monitor that is not operational names one.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
For a vm-disk, WorkloadsReady is the one condition that reports
whether CDI populated the disk, and its generic "One or more workloads
are not operational" gives an operator nothing to act on: a disk has
no workloads, and the stuck DataVolume and its phase are exactly what
they need to know.
The WorkloadMonitor status gains a message that the controller fills
with each selected DataVolume that is not ready and its phase, and the
Application API uses the messages of the monitors that are not
operational in WorkloadsReady instead of the generic text whenever one
of them names a cause.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A tlsPassthroughServices entry publishes a hostname on port 443, which
suits a service whose clients speak HTTP and mis-serves one whose
clients expect a database on its own port. The new field takes a name,
a native port and the SNI hostname to answer on it, so an engine can be
reached where its drivers already look.
The cross-field rules are enforced at admission as well as in the
controller. renderGateway is the first thing that renders, so a spec
the apiserver accepts and the renderer refuses aborts everything behind
it, and one mistyped hostname would freeze the Gateway and every route
status where they stand. The controller keeps its own copy because it
and the CRD roll out separately. Rules with no admission form, and why
each one has none, are named on the type.
tlsPassthroughServices and apex gain bounds at the same time, for the
CEL cost estimator rather than for their own sake: the new rules
concatenate apex and scan the services list, and an unbounded string or
list makes the install-time estimate exceed the per-CRD budget and the
apiserver refuse the whole definition. Every bound is a kind the
apiserver ratchets, so a stored value that violates one keeps accepting
writes that leave it alone.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The controller-owned redirect route carried no spec.hostnames, relying
on the port-80 listener to match every host. Name the tenant apex and
its wildcard instead.
A route in a tenant namespace has to declare hostnames inside its own
apex to pass the route-hostname admission policy, and a route that
names none has nothing to compare against that apex. Naming them also
states the coverage the route wants rather than inheriting whatever its
listener happens to carry.
Both hostnames are required: a wildcard is a suffix match over any
number of labels, so "*.<apex>" covers every subdomain at every depth
but not the bare "<apex>".
Coverage narrows in one direction worth stating: a request arriving on
port 80 with a Host outside the tenant apex now falls through to a 404
instead of a 301.
Skip this route when collecting hostname claims. Before it carried
hostnames it contributed nothing to that set, and letting its hostnames
through instead provisions listeners and certificates for them, which
fails. The check matches on name and controller ownership, not on the
labels the renderer stamps, since those are writable by anyone who can
create a route.
Deriving hostnames from spec.apex makes that field load-bearing for
admission, so it gains MinLength=1 and a matching check in the renderer
for clusters whose CRD is older than the binary.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Some GatewayClasses have their provider terminate TLS in front of the
Gateway and reach it over plain HTTP. On such a class the platform
issued origin certificates nobody serves, and where ACME could not
complete, the Gateway's listeners never became ready.
Add certMode=edge to TenantGateway. The controller renders the apex,
its wildcard and every inheriting child apex as plain HTTP listeners
carrying the same label-based allowedRoutes as the HTTPS listeners of
the other modes, so application HTTPRoutes attach by hostname
unchanged. It mints no Issuer and no Certificate, and removes the
http-to-https redirect route, because the edge performs that redirect
itself. TLSPassthroughServices are not rendered: an HTTP-only edge
cannot carry a TLS listener, which leaves the endpoints relying on one
unpublished through such a Gateway.
The hostname-less port-80 listener is left out. It carries the ACME
challenge and the redirect in the other modes and has neither here, so
it would only be a catch-all admitting any hostname. Leaving it out
also keeps a hostname-pinned listener at index 0, which is what
Cilium's first-listener-wins namespace check reads before the fix in
cilium/cilium#45693.
Moving into the mode garbage-collects the Issuer and Certificate
objects the controller owned. The TLS Secrets they issued stay behind:
cert-manager runs with enableCertificateOwnerRef false, so a
Certificate does not own the Secret it issued into. Every mode
transition already behaves that way.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
Every catalog engine issues a per-release CA, but the object holding
ca.crt usually holds a private key beside it (cert-manager's tls.key,
CloudNativePG's ca.key), so granting a tenant read on it hands over key
material too. Add a controller that publishes the trust anchor as one
canonical, key-free Secret per release, "<release>.tenant-ca", holding
only ca.crt.
A chart declares the anchor by rendering a namespaced TenantProjection
sentinel (internal.cozystack.io/v1alpha1) that names the source Secret
and key. The controller rebuilds ca.crt from the parsed certificate DER
rather than copying the source, refuses any value carrying private key
material, and owns the projection by owner reference so a foreign Secret
at the same name is never overwritten. No tenant RBAC grants any verb on
internal.cozystack.io; a ValidatingAdmissionPolicy pins writes to
helm-controller as defence in depth.
The controller does not grant tenant read access — the lineage webhook
does, from the ApplicationDefinition's spec.secrets. This side stamps the
engine-agnostic internal.cozystack.io/tenant-ca label and forces a
re-admission when those selectors change.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Remove the network.cozystack.io API group (ExposureClass + ServiceExposure)
and its reconciler. Managed-application charts own their Service, so native
type: LoadBalancer + loadBalancerClass + an admin-provisioned pool covers
external exposure without a dedicated API group and controller. The host
ingress -- the only consumer -- keeps rendering type: LoadBalancer and gains
an optional publishing.loadBalancerClass to pick the LB controller; the
externalIPs (node-IP) default path is unchanged.
Rationale in cozystack#3164.
Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Introduce a new network.cozystack.io/v1alpha1 API group with two kinds
that form the mechanism-agnostic LoadBalancer exposure model:
- ExposureClass (cluster-scoped): the admin binds a logical class to a
concrete backend (externalIPs, metallb, cilium, robotlb) and its VIP
address scope, following the StorageClass / IngressClass pattern.
- ServiceExposure (namespaced): the tenant-facing intent to expose a
Service externally via a named class, with status reporting the
resolved backend and assigned IPs.
Wire api/network into the controller-gen CRD generation so deepcopy and
the two CRD manifests are produced alongside the existing groups.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Operators who already hold a wildcard TLS certificate (purchased, or
issued by a corporate CA) had no way to make a tenant Gateway serve
under it: the only cert modes were http01 (per-listener ACME) and
dns01 (controller-minted wildcard via ACME).
Add a third mode, existingSecret, that references a pre-existing TLS
Secret in the TenantGateway's own namespace via spec.wildcardSecretRef.
In this mode the controller mints no ACME Issuer and no Certificate;
the wildcard, apex, and per-child-apex HTTPS listeners reference the
supplied Secret directly. A missing wildcardSecretRef fails fast with
Ready=False rather than producing a Gateway pointing at a nonexistent
Secret. Switching into existingSecret from http01/dns01 reclaims the
now-unused Issuer and Certificates so no ACME machinery is left behind.
The non-cert listeners (http->https redirect, TLS passthrough) are
unaffected and continue to render.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.
The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.
publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.
Security model -- seven independent layers, all admission-time gates
fail-closed:
1. Listener allowedRoutes namespace whitelist via the
kube-apiserver-written kubernetes.io/metadata.name label;
plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
passthrough listeners restrict allowedRoutes.kinds to
HTTPRoute / TLSRoute respectively.
2. cozystack-gateway-hostname-policy VAP on Gateway.
3. cozystack-gateway-attached-namespaces-policy VAP on Package --
blocks tenant-* in gateway.attachedNamespaces.
4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
writes from non-trusted callers.
5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
writes to namespace.cozystack.io/host from non-trusted callers.
6. Render-time fail in cozystack-basics for tenant-* in
gateway.attachedNamespaces.
7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
scoped to tenant-* namespaces.
Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).
Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.
Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.
Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.
Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.
Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.
Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The dashboard controller used to project ApplicationDefinitions
into a set of dashboard.cozystack.io CRDs (Sidebar, Factory,
Navigation, Breadcrumb, CustomFormsOverride, CustomFormsPrefill,
CustomColumnsOverride, MarketplacePanel, TableUriMapping,
CFOMapping, BreadcrumbInside) consumed by the legacy UI.
cozystack-ui now reads ApplicationDefinitions directly, so the
controller, its API types, the generated CRDs and the related
RBAC rules are no longer used.
Removes:
- internal/controller/dashboard
- api/dashboard/v1alpha1
- packages/system/cozystack-controller/definitions/dashboard.cozystack.io_*.yaml
- dashboard.cozystack.io wiring from cmd/cozystack-controller/main.go
- dashboard.cozystack.io RBAC rules from cozystack-controller and cozy-dashboard charts
- ./api/dashboard/... codegen path from hack/update-codegen.sh
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
- Upgrade CRDs in CozyStack dashboard controller
- Add Ingress proxy timeouts for WebSocket to work without terminations
- Add CFOMapping custom resource
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Aleksei Sviridkin