Files
Aleksei Sviridkin 2bf9ae4a27 feat(gateway)!: serve passthrough hostnames and stop terminating them
Each tlsPassthroughListeners entry renders a mode=Passthrough listener
on its own port, answering the entry's hostname by SNI and admitting
TLSRoutes from the publishing tenant's namespace alone. It declares no
allowedRoutes.kinds: at the pinned Cilium v1.19.5 a listener that
spells its kinds out has them applied to every route on the Gateway
rather than to itself, so naming TLSRoute here would reject every
HTTPRoute on the Gateway.

A hostname a passthrough listener answers no longer earns an
HTTPS-terminate listener or a Gateway-issued certificate. That is the
break, and it is keyed on service rather than on declaration: the same
Cilium builds a passthrough filter chain per route out of the backends
that survived resolution, so a listener no TLSRoute attaches to, and a
route whose backendRefs resolve to nothing, both match no ClientHello
and leave the terminate listener holding the name. Eligibility asks
what v1.19.5 asks and no more, the reference allowed and the Service
present, and fails closed on a read it cannot complete. Services and
ReferenceGrants are watched, so a withdrawal deferred on a missing
backend lands on the pass that sees it arrive, and the Service watch
takes only the creates and deletes, existence being all that is read.

Which hostname a route claims is answered the same way. spec.hostnames
is optional on a TLSRoute, and ComputeHosts substitutes the hostname of
each TLS listener the route's parentRefs select for a route that
declares none, so such a route carries the SNI and takes the name over
exactly as one that spells it out. A declared hostname broader than the
listener's is that substitution from the other side: the route is
served on the listener's name and not on the wildcard it wrote, so a
claim covering a reserved name counts as a claim on it.

The substitution reaches only the listeners that would admit the
route. A native-port listener takes the publishing tenant's namespace
alone, so Cilium never puts a route from elsewhere on it and never
serves that route on its hostname; borrowing the name anyway refuses
the route for a namespace, over a hostname its own object does not
carry, in the same pass that sheds a terminate listener because that
same route is serving a name it borrowed legitimately. A route that
names the listener in its sectionName keeps the borrow, since there
the refusal is the answer to what it asked for.

A route the Gateway renders no listener for hears it from this
controller instead of from Cilium alone: the hostname a passthrough
listener took, the hostname nothing answers, the sectionName or port
naming a listener that answers something else, the sectionName naming
no rendered passthrough listener, and the native-port listener that
admits the tenant alone, each with its own Gateway API reason and a
message naming the hostnames that lost. An HTTPRoute pinning a
rendered passthrough listener by sectionName is refused there too,
since such a listener forwards a stream it never terminates and takes
TLSRoute alone by protocol. Nothing else tells that route: the port-443
listeners name HTTPRoute in their kinds to keep every port-443 set
uniform, which the Gateway reports back as InvalidRouteKinds rather
than as a grant, and the pin reads a listener's kinds against every
route on the Gateway, so it reports the route accepted.

The field is confined to certMode http01, stated as the modes that can
carry it so one added later is refused until someone rules on it. The
wildcard modes cover the whole apex with one terminate listener the
pinned Cilium cannot keep apart from a passthrough listener beneath it,
and edge terminates TLS at the class provider and renders no TLS
listener at all.

An HTTPRoute claiming a wildcard is refused under http01 before the
race and the overlap are read. The HTTP-01 challenge issues no
wildcard certificate, so the terminate listener rendered for the claim
could never carry the name, and the listener name built from the
hostname's first label carried the asterisk, which the Gateway CRD
admits in a hostname and refuses in a name, so the apiserver refused
the whole Gateway write and every other listener on it froze where it
stood. The renderer now writes that label as "wildcard", so every
hostname the CRD admits yields a name it admits, and no name that was
already admissible moves.

That refusal is a statement about the terminate listener, so it is
judged where that listener is decided rather than ahead of everything
else. The port-80 listener carries no hostname and serves a wildcard
claim as readily as any other, so a route that selects it, by pinning
the port or by naming the listener, never needed the listener HTTP-01
cannot supply and is served rather than refused. Judged first, the
refusal reached that route too and reported it unserved while Cilium
was serving it in plaintext.

A parentRef port on an HTTPRoute is read the way servableOn reads it
for a TLSRoute. The terminate listener sits on 443, so a route pinning
another port attaches to no listener the render would give it, and it
used to earn a listener and an ACME certificate for a route that
attached to nothing. Port 80 is the plain-HTTP listener, which admits
the publishing tenant alone: a route from there pinning it is served
plain and earns neither a listener nor a refusal.

A route status write that fails is told apart by kind. A conflict or
a route that vanished between the list and the write is a race between
the two status writers this Gateway has: the reconcile requeues and
retries the write without pinning Ready=False on the tenant, and the
status pass runs after the http-to-https redirect, so such a race
cannot withhold the one object that guard exists to provide. A write
the apiserver will refuse on every pass still fails the reconcile,
after every other route has been written, because the condition is the
only object saying why a hostname is not served, and a pass that
reported success over a lost write left the route reading Accepted=True
until the resync.

The retryable half is logged on the way out. Declining to pin
Ready=False leaves the requeue as the only thing acting on it, and the
requeue says nothing: the tenant goes on reading Ready, no event is
recorded, and returning no error opts out of the backoff that would
widen the interval, so a write failing on every pass retries at a fixed
period indefinitely with nothing anywhere to show for it.

A TLSRoute claiming a hostname broader than the listener's is pulled
into the eligibility question for each name it covers, because the
pinned Cilium serves it on the listener's own name. That is right only
where the listener answering a name is named exactly that name. Under
a wildcard entry the substitution hands such a route the wildcard, so
its chain carries "*.db.<apex>" while the terminate chain carries the
exact name beneath, and Envoy matches an exact server name ahead of a
wildcard, so nothing collides; pulling the route in there withdrew a
served endpoint and handed its name to the database behind the
passthrough listener. The pull-in is now keyed on an exact listener
match.

Every condition this controller writes on an HTTPRoute, and the one it
writes on a TLSRoute declaring no hostnames, is written under http01,
and the whole-apex modes collect no claims, so a switch of certMode
left those verdicts standing over listeners the new mode does not
render. The whole-apex pass now drops this controller's entry from
HTTPRoutes and from hostname-less TLSRoutes; the edge refusal stays
confined to routes that declare hostnames, since a refusal written on
a lent hostname would have no path back.

Within http01 the same verdicts outlive their listeners for a second
reason, and no mode change is needed to reach it. The status pass
walks the routes that attached rather than the ones that claimed a
hostname, and retracts its entry from the difference, because a claim
set is not a route's own property once a hostname can be borrowed: a
TLSRoute declaring none takes the hostname of every passthrough
listener its parentRef selects, so removing that listener empties the
claim set while the route stands unchanged, and an HTTPRoute reaches
the same state by dropping its hostnames. Judged on claims alone such
a route keeps whatever the last pass that did judge it left behind,
describing a listener the Gateway no longer has.

An HTTPRoute pinned by sectionName to a passthrough listener this
Gateway does not render is refused with NoMatchingParent, the reading
a TLSRoute in the same shape already got, instead of earning a
terminate listener it could never attach to.

A route served on the port-80 listener, by parentRef port or by
naming it, earns no terminate listener and is refused nothing, but it
publishes the name from its namespace all the same, so it stays in the
hostname race. Every recount now ranks it with the field it recounts,
because a recount that left it out crowned the namespace that had lost
to it and dropped a loss the first count recorded: a third route
refused in a third namespace, or a passthrough entry declared for the
name, flipped the loser to Accepted=True.

Selecting the port-80 listener by name is judged as selecting it by
port: from a namespace the listener admits the route is served plain,
from anywhere else the listener turns it away and the route reads
NotAllowedByListeners, and a sectionName and a parentRef port that
select different listeners select none and read NoMatchingParent,
which is what Gateway API requires when both are given. Before this a
route naming the listener from a refused namespace earned a terminate
listener and a certificate it could never attach to.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-10 18:47:30 +03:00

361 lines
15 KiB
Go

/*
Copyright 2025.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package main
import (
"crypto/tls"
"flag"
"os"
"time"
// Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.)
// to ensure that exec-entrypoint and run can make use of them.
_ "k8s.io/client-go/plugin/pkg/client/auth"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/runtime"
utilruntime "k8s.io/apimachinery/pkg/util/runtime"
clientgoscheme "k8s.io/client-go/kubernetes/scheme"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/cache"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/cluster"
"sigs.k8s.io/controller-runtime/pkg/healthz"
"sigs.k8s.io/controller-runtime/pkg/log/zap"
"sigs.k8s.io/controller-runtime/pkg/metrics/filters"
metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
"sigs.k8s.io/controller-runtime/pkg/webhook"
gatewayv1alpha1 "github.com/cozystack/cozystack/api/gateway/v1alpha1"
internalv1alpha1 "github.com/cozystack/cozystack/api/internalapi/v1alpha1"
cozystackiov1alpha1 "github.com/cozystack/cozystack/api/v1alpha1"
"github.com/cozystack/cozystack/internal/controller"
"github.com/cozystack/cozystack/internal/controller/cacert"
"github.com/cozystack/cozystack/internal/controller/tenantgateway"
"github.com/cozystack/cozystack/internal/controller/tenantquota"
"github.com/cozystack/cozystack/internal/controller/wildcardsecret"
"github.com/cozystack/cozystack/internal/telemetry"
cmv1 "github.com/cert-manager/cert-manager/pkg/apis/certmanager/v1"
helmv2 "github.com/fluxcd/helm-controller/api/v2"
cosiv1alpha1 "sigs.k8s.io/container-object-storage-interface-api/apis/objectstorage/v1alpha1"
gatewayv1 "sigs.k8s.io/gateway-api/apis/v1"
gatewayv1alpha2 "sigs.k8s.io/gateway-api/apis/v1alpha2"
gatewayv1beta1 "sigs.k8s.io/gateway-api/apis/v1beta1"
// +kubebuilder:scaffold:imports
)
var (
scheme = runtime.NewScheme()
setupLog = ctrl.Log.WithName("setup")
)
func init() {
utilruntime.Must(clientgoscheme.AddToScheme(scheme))
utilruntime.Must(cozystackiov1alpha1.AddToScheme(scheme))
utilruntime.Must(gatewayv1alpha1.AddToScheme(scheme))
utilruntime.Must(internalv1alpha1.AddToScheme(scheme))
utilruntime.Must(gatewayv1.Install(scheme))
utilruntime.Must(gatewayv1alpha2.Install(scheme))
utilruntime.Must(gatewayv1beta1.Install(scheme))
utilruntime.Must(cmv1.AddToScheme(scheme))
utilruntime.Must(helmv2.AddToScheme(scheme))
utilruntime.Must(cosiv1alpha1.AddToScheme(scheme))
// +kubebuilder:scaffold:scheme
}
func main() {
var metricsAddr string
var enableLeaderElection bool
var probeAddr string
var secureMetrics bool
var enableHTTP2 bool
var disableTelemetry bool
var telemetryEndpoint string
var telemetryInterval string
var quotaBufferPercent int64
var seaweedfsMetricsEndpoint string
var tlsOpts []func(*tls.Config)
flag.StringVar(&metricsAddr, "metrics-bind-address", "0", "The address the metrics endpoint binds to. "+
"Use :8443 for HTTPS or :8080 for HTTP, or leave as 0 to disable the metrics service.")
flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
flag.BoolVar(&enableLeaderElection, "leader-elect", false,
"Enable leader election for controller manager. "+
"Enabling this will ensure there is only one active controller manager.")
flag.BoolVar(&secureMetrics, "metrics-secure", true,
"If set, the metrics endpoint is served securely via HTTPS. Use --metrics-secure=false to use HTTP instead.")
flag.BoolVar(&enableHTTP2, "enable-http2", false,
"If set, HTTP/2 will be enabled for the metrics and webhook servers")
flag.BoolVar(&disableTelemetry, "disable-telemetry", false,
"Disable telemetry collection")
flag.StringVar(&telemetryEndpoint, "telemetry-endpoint", "https://telemetry.cozystack.io",
"Endpoint for sending telemetry data")
flag.StringVar(&telemetryInterval, "telemetry-interval", "15m",
"Interval between telemetry data collection (e.g. 15m, 1h)")
flag.Int64Var(&quotaBufferPercent, "tenant-quota-buffer-percent", 0,
"Temporary buffer (e.g. 130 = +30%) added to every hierarchical tenant quota pool so workloads already over a freshly-introduced quota keep running during rollout. 0 disables it.")
flag.StringVar(&seaweedfsMetricsEndpoint, "seaweedfs-metrics-endpoint", "",
"Base URL of a Prometheus-compatible query API to fetch SeaweedFS bucket size metrics from, e.g. https://vm.example.com/path/to/prometheus (/api/v1/query is appended). "+
"Overrides discovery via the namespace.cozystack.io/monitoring label; use when SeaweedFS and the monitoring stack that scrapes it run in a separate cluster. "+
"Basic auth may be embedded as userinfo (https://user:pass@host/...). Empty keeps label-based discovery.")
opts := zap.Options{
Development: false,
}
opts.BindFlags(flag.CommandLine)
flag.Parse()
// Parse telemetry interval
interval, err := time.ParseDuration(telemetryInterval)
if err != nil {
setupLog.Error(err, "invalid telemetry interval")
os.Exit(1)
}
// Validate and normalize the SeaweedFS metrics endpoint override; the sizes
// it serves feed billing, so a malformed URL must fail startup, not queries.
if seaweedfsMetricsEndpoint != "" {
seaweedfsMetricsEndpoint, err = controller.ParseMetricsEndpointURL(seaweedfsMetricsEndpoint)
if err != nil {
setupLog.Error(err, "invalid --seaweedfs-metrics-endpoint")
os.Exit(1)
}
}
// Configure telemetry
telemetryConfig := telemetry.Config{
Disabled: disableTelemetry,
Endpoint: telemetryEndpoint,
Interval: interval,
}
ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))
// if the enable-http2 flag is false (the default), http/2 should be disabled
// due to its vulnerabilities. More specifically, disabling http/2 will
// prevent from being vulnerable to the HTTP/2 Stream Cancellation and
// Rapid Reset CVEs. For more information see:
// - https://github.com/advisories/GHSA-qppj-fm5r-hxr3
// - https://github.com/advisories/GHSA-4374-p667-p6c8
disableHTTP2 := func(c *tls.Config) {
setupLog.Info("disabling http/2")
c.NextProtos = []string{"http/1.1"}
}
if !enableHTTP2 {
tlsOpts = append(tlsOpts, disableHTTP2)
}
webhookServer := webhook.NewServer(webhook.Options{
TLSOpts: tlsOpts,
})
// Metrics endpoint is enabled in 'config/default/kustomization.yaml'. The Metrics options configure the server.
// More info:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.19.1/pkg/metrics/server
// - https://book.kubebuilder.io/reference/metrics.html
metricsServerOptions := metricsserver.Options{
BindAddress: metricsAddr,
SecureServing: secureMetrics,
TLSOpts: tlsOpts,
}
if secureMetrics {
// FilterProvider is used to protect the metrics endpoint with authn/authz.
// These configurations ensure that only authorized users and service accounts
// can access the metrics endpoint. The RBAC are configured in 'config/rbac/kustomization.yaml'. More info:
// https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.19.1/pkg/metrics/filters#WithAuthenticationAndAuthorization
metricsServerOptions.FilterProvider = filters.WithAuthenticationAndAuthorization
// TODO(user): If CertDir, CertName, and KeyName are not specified, controller-runtime will automatically
// generate self-signed certificates for the metrics server. While convenient for development and testing,
// this setup is not recommended for production.
}
// Configure rate limiting for the Kubernetes client
config := ctrl.GetConfigOrDie()
config.QPS = 50.0 // Increased from default 5.0
config.Burst = 100 // Increased from default 10
mgr, err := ctrl.NewManager(config, ctrl.Options{
Scheme: scheme,
Metrics: metricsServerOptions,
WebhookServer: webhookServer,
HealthProbeBindAddress: probeAddr,
LeaderElection: enableLeaderElection,
LeaderElectionID: "19a0338c.cozystack.io",
// Scope the shared Secret informer so the WildcardSecret reconciler's
// cluster-wide Secret watch does not cache every Secret (and its key
// material) in memory. Only managed wildcard replicas and the values
// channel are cached; no other reconciler reads Secrets through THIS
// (the manager's) typed Secret cache. See wildcardsecret.SecretCacheByObject.
//
// The CA-extraction reconciler does NOT use this cache at all — neither its
// typed Secret informer nor a metadata one. A metadata informer for
// v1/Secret would route to THIS same per-GVK cache and inherit its
// wildcard-replica label selector, so its source watch would never fire for
// an unlabelled CA source. It therefore owns a SEPARATE metadata-only cache
// (caSecretCluster, below) and reads the one source it projects through the
// uncached APIReader. So this manager-level Secret scoping is
// wildcardsecret's alone.
Cache: cache.Options{
ByObject: map[client.Object]cache.ByObject{
&corev1.Secret{}: wildcardsecret.SecretCacheByObject(),
},
},
// LeaderElectionReleaseOnCancel defines if the leader should step down voluntarily
// when the Manager ends. This requires the binary to immediately end when the
// Manager is stopped, otherwise, this setting is unsafe. Setting this significantly
// speeds up voluntary leader transitions as the new leader don't have to wait
// LeaseDuration time first.
//
// In the default scaffold provided, the program ends immediately after
// the manager stops, so would be fine to enable this option. However,
// if you are doing or is intended to do any operation such as perform cleanups
// after the manager stops then its usage might be unsafe.
// LeaderElectionReleaseOnCancel: true,
})
if err != nil {
setupLog.Error(err, "unable to start manager")
os.Exit(1)
}
if err = (&controller.WorkloadMonitorReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
Recorder: mgr.GetEventRecorderFor("workloadmonitor-controller"),
SeaweedfsMetricsEndpoint: seaweedfsMetricsEndpoint,
}).SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "WorkloadMonitor")
os.Exit(1)
}
if err = (&controller.WorkloadReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
}).SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "WorkloadReconciler")
os.Exit(1)
}
if err = (&controller.ApplicationDefinitionReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
}).SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "ApplicationDefinitionReconciler")
os.Exit(1)
}
if err = (&controller.ApplicationDefinitionHelmReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
}).SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "ApplicationDefinitionHelmReconciler")
os.Exit(1)
}
if err = (&tenantgateway.Reconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
}).SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "TenantGateway")
os.Exit(1)
}
if err = (&tenantquota.Reconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
Recorder: mgr.GetEventRecorderFor("tenantquota-controller"),
BufferPercent: quotaBufferPercent,
}).SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "TenantQuota")
os.Exit(1)
}
if err = (&wildcardsecret.Reconciler{
Client: mgr.GetClient(),
Reader: mgr.GetAPIReader(),
Scheme: mgr.GetScheme(),
Recorder: mgr.GetEventRecorderFor("wildcardsecret"),
}).SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "WildcardSecret")
os.Exit(1)
}
// The CA-extraction reconciler watches its source Secrets as metadata only, so
// no private key ever enters a cache. It cannot watch them on the manager's
// cache: that cache's Secret informer is label-scoped to WildcardSecret's
// replicas (above), a metadata informer for the same v1/Secret GVK routes to
// that same scoped informer and inherits the selector, and the watch would
// then silently never fire for an unlabelled CA source — the projection would
// appear only on the slow resync. So it owns a SEPARATE cache, metadata-only
// and unscoped, holding Secret metadata stubs cluster-wide (no key material).
// It is a cluster.Cluster so mgr.Add registers it in the manager's
// cache-runnable group — started, waited-for-sync before the controllers run,
// and stopped on shutdown, exactly like the shared cache.
caSecretCluster, err := cluster.New(config, func(o *cluster.Options) {
o.Scheme = scheme
})
if err != nil {
setupLog.Error(err, "unable to build the CA source metadata cache")
os.Exit(1)
}
if err = mgr.Add(caSecretCluster); err != nil {
setupLog.Error(err, "unable to add the CA source metadata cache to the manager")
os.Exit(1)
}
if err = (&cacert.Reconciler{
Client: mgr.GetClient(),
Reader: mgr.GetAPIReader(),
Recorder: mgr.GetEventRecorderFor("cacert-controller"),
}).SetupWithManager(mgr, caSecretCluster.GetCache()); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "CACert")
os.Exit(1)
}
// +kubebuilder:scaffold:builder
if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
setupLog.Error(err, "unable to set up health check")
os.Exit(1)
}
if err := mgr.AddReadyzCheck("readyz", healthz.Ping); err != nil {
setupLog.Error(err, "unable to set up ready check")
os.Exit(1)
}
// Initialize telemetry collector
collector, err := telemetry.NewCollector(mgr.GetClient(), &telemetryConfig, mgr.GetConfig())
if err != nil {
setupLog.V(1).Error(err, "unable to create telemetry collector, telemetry will be disabled")
}
if collector != nil {
if err := mgr.Add(collector); err != nil {
setupLog.Error(err, "unable to set up telemetry collector")
setupLog.V(1).Error(err, "unable to set up telemetry collector, continuing without telemetry")
}
}
setupLog.Info("starting manager")
if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
setupLog.Error(err, "problem running manager")
os.Exit(1)
}
}