feat(observability-proxy): publish the proxy on its own host (#5267)

<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Adds an optional public endpoint for the observability proxy, so that
clients outside the cluster (kubectl, scripts, third-party tools) can
call it directly with their Kubernetes bearer token. The existing paths
do not deliver the token: kube-apiserver removes the `Authorization`
header after it authenticates a request, so the service proxy of
kube-apiserver cannot carry it, and the console route goes through the
dashboard gatekeeper, which with OIDC enabled answers a client without a
session cookie with 401 and exists only where the dashboard is
published.

The endpoint is off by default. It is published at
`observability-proxy.<publishing.host>` once `observability-proxy` is
added to `publishing.exposedServices`, in the same two modes the
dashboard uses:

- `gateway.enabled: false`: an Ingress on the publishing ingress class,
with a per-host cert-manager certificate, or the operator's wildcard
certificate when `publishing.certificates.wildcardSecretName` is set
(the TLS section stays, so HTTP is still redirected to HTTPS).
- `gateway.enabled: true`: an HTTPRoute on the publishing tenant's
Gateway. The platform now lists `cozy-observability-proxy` in
`gateway.attachedNamespaces` so that the Gateway admits the route. A
cluster that overrides `gateway.attachedNamespaces` replaces the whole
list and has to add the namespace itself.

In Ingress mode the request crosses the publishing tenant's egress
policies, which deny everything except the `allow-to-*` destinations, so
the tenant chart gains `allow-to-observability-proxy`. It matches the
whole `cozy-observability-proxy` namespace, as `allow-to-linstor-gui`
does, because cert-manager runs the HTTP-01 solver pods for the proxy's
certificate there.

The route passes the request through unchanged: no path rewrite and no
authentication at the edge, so `/apis/...` and the bearer token reach
the proxy as sent. The edge limits are set so that they never end a
request before the proxy does:

- The tenant ingress-nginx reads upstream with a 10s timeout by default,
while the proxy allows a non-tail request up to `--request-timeout` (1m)
and keeps a log tail open until it has been silent for
`--tail-idle-timeout`. Before the first byte of a response, the token
and access reviews can add up to another minute. The Ingress sets a 360s
read timeout and keeps response buffering off, so tail lines reach the
client as they arrive.
- The Ingress sets a 1 MiB body limit, the same limit the proxy enforces
on a request body.
- On an HTTPRoute that sets no timeout, Cilium keeps Envoy's 15s route
timeout, which would cut every tail. The route disables it
(`timeouts.request: 0s`), because the proxy limits each request itself.
Envoy's stream idle timeout (Cilium
`envoy.streamIdleTimeoutDurationSeconds`, 300s by default) still applies
and cannot be changed from an HTTPRoute, so the chart now runs the proxy
with `--tail-idle-timeout=4m30s` instead of the 5m default. A tail that
stops sending lines is then ended by the proxy as a complete response,
not reset by Envoy. Before the first byte, slow token and access reviews
together with a log store that never sends headers can still exceed
300s; that case ends in an error either way.

The proxy never reads the client address or `X-Forwarded-*`. All its
limits and its logs use the authenticated user, so nothing changes
behind an ingress and the client cannot spoof anything through headers.

### Security

This adds a new internet-facing entry point when it is enabled. What
protects it:

- Every request to a tenant root needs a bearer token that
kube-apiserver accepts through a TokenReview, and a SubjectAccessReview
for the requested namespace. Paths the proxy does not serve return 404
before any review, and a request without a bearer token returns 401
without a review.
- Per-user limits apply after authentication: request rate, concurrent
requests, concurrent tails per user, and a total cap on open tails.
Every request is bounded by the request timeout, and every tail by the
idle timeout.
- The body is capped at 1 MiB at the edge and in the proxy.
- The new tenant egress policy lets every pod of a tenant open a
connection to the proxy namespace, whether or not the proxy is
published. This gives a tenant workload nothing it cannot already do
through the published host, and the proxy asks for a token in both
cases.

What it does not protect against: an unauthenticated client can make the
proxy send one TokenReview per distinct token it invents. Rejected
tokens are cached by hash for the auth cache TTL, so a repeated token
costs nothing more. This is the same cost a client can already cause by
calling the published Kubernetes API (`api` is in
`publishing.exposedServices` by default). No per-IP rate limit is added
at the edge, because with `publishing.externalIPs` the ingress Service
uses `externalTrafficPolicy: Cluster`, which can hide client addresses
behind node addresses, so a per-IP limit would throttle all clients
together.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not a UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#787
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff changes `packages/core/platform/values.yaml`, which reaches
cozystack/website:
`content/en/docs/next/operations/configuration/platform-package.md`
lists the possible values of `publishing.exposedServices` and the
default `gateway.attachedNamespaces` list, and both need the new
entries. The observability-proxy documentation is tracked in
cozystack/website#787, which does not cover the public endpoint yet; the
endpoint and the new list entries will be added there once this PR is
accepted. No other repository in the trigger map is reached by this
diff.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(observability-proxy): the observability proxy can be published at observability-proxy.<host> for clients outside the cluster that use a Kubernetes bearer token. It is off by default; add `observability-proxy` to `publishing.exposedServices` to publish it, through an Ingress or, with `gateway.enabled`, an HTTPRoute. Clusters that override `gateway.attachedNamespaces` must add `cozy-observability-proxy` to it for the Gateway mode. Tenants gain an egress policy to the proxy namespace, which the Ingress mode needs.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
  * Tenant workloads can now connect to the observability proxy.
* The observability proxy can be published through an Ingress or Gateway
route when enabled in the service exposure settings.
* Proxy routing supports long-running requests and streams with adjusted
timeout handling.
* Ingress publishing supports per-host or wildcard certificates and
configurable certificate challenge settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
Aleksei Sviridkin authored and GitHub committed 2026-11-13 20:41:32 +03:00
commit 53078a1d29
9 files changed
+381

No files matched your search

@@ -259,6 +259,20 @@ spec:
- ports:
- port: "8443"
protocol: TCP
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
# The whole namespace, as for linstor-gui: the ingress controller also
# has to reach the ACME HTTP-01 solver pods cert-manager starts there.
name: allow-to-observability-proxy
namespace: {{ include "tenant.name" . }}
spec:
endpointSelector: {}
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": cozy-observability-proxy
{{- if .Values.monitoring }}
---
apiVersion: cilium.io/v2
@@ -0,0 +1,64 @@
suite: tenant egress to the observability proxy
# The observability-proxy Ingress in cozy-observability-proxy is served by
# the ingress controller running in tenant-root. Tenant egress is
# default-deny plus explicit allow-to-* policies, so without this one the
# controller reaches neither the proxy nor the ACME HTTP-01 solver pod that
# cert-manager starts in the same namespace; hence the whole namespace, as
# allow-to-linstor-gui does.
templates:
- templates/networkpolicy.yaml
tests:
- it: lets every pod of the root tenant reach the proxy namespace
release:
name: tenant-root
namespace: tenant-root
asserts:
- documentSelector:
path: metadata.name
value: allow-to-observability-proxy
isKind:
of: CiliumNetworkPolicy
- documentSelector:
path: metadata.name
value: allow-to-observability-proxy
equal:
path: metadata.namespace
value: tenant-root
- documentSelector:
path: metadata.name
value: allow-to-observability-proxy
equal:
path: spec
value:
endpointSelector: {}
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": cozy-observability-proxy
- it: renders in a nested tenant with every module off
release:
name: tenant-bar
namespace: tenant-foo
set:
monitoring: false
ingress: false
etcd: false
asserts:
- documentSelector:
path: metadata.name
value: allow-to-observability-proxy
equal:
path: metadata.namespace
value: tenant-foo-bar
- it: renders nothing where the tenant Cilium policies are off
release:
name: tenant-root
namespace: tenant-root
set:
_cluster:
bundle-name: isp-hosted
asserts:
- hasDocuments:
count: 0
@@ -5,6 +5,7 @@ suite: observability-proxy package wiring
templates:
- templates/sources.yaml
- templates/bundles/system.yaml
- templates/apps.yaml
release:
name: cozystack
namespace: cozy-system
@@ -67,3 +68,12 @@ tests:
kind: Package
name: cozystack.observability-proxy
any: true
# The proxy's HTTPRoute lives in its own namespace, and the tenant Gateway
# admits routes only from the namespaces listed here.
- it: lets the proxy namespace attach a route to the publishing Gateway
template: templates/apps.yaml
asserts:
- matchRegex:
path: stringData["values.yaml"]
pattern: 'gateway-attached-namespaces: "([^"]*,)?cozy-observability-proxy(,[^"]*)?"'
+1
View File
@@ -668,6 +668,7 @@ gateway:
- cozy-kubevirt-cdi
- cozy-monitoring
- cozy-linstor-gui
- cozy-observability-proxy
- default
# Platform ingress configuration.
ingress:
@@ -38,6 +38,10 @@ spec:
- --cluster-domain={{ $clusterDomain }}
- --root-tenant={{ .Values.observabilityProxy.rootTenant }}
- --auth-cache-ttl={{ .Values.observabilityProxy.authCacheTTL }}
{{- /* Below the 300s stream idle timeout of a Cilium Gateway, so a
published tail that stops sending lines is ended by the proxy,
not reset by Envoy. */}}
- --tail-idle-timeout=4m30s
- --metrics-shortterm-store={{ .Values.observabilityProxy.metrics.shorttermStore }}
- --metrics-longterm-store={{ .Values.observabilityProxy.metrics.longtermStore }}
- --metrics-shortterm-retention={{ .Values.observabilityProxy.metrics.shorttermRetention }}
@@ -0,0 +1,30 @@
{{- $cluster := .Values._cluster | default dict }}
{{- $host := index $cluster "root-host" }}
{{- $exposeServices := splitList "," ((index $cluster "expose-services") | default "") }}
{{- $exposeIngress := (index $cluster "expose-ingress") | default "tenant-root" }}
{{- $gatewayEnabled := (index $cluster "gateway-enabled") | default "false" }}
{{- if and (has "observability-proxy" $exposeServices) (eq $gatewayEnabled "true") }}
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: observability-proxy
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: cozystack
namespace: {{ $exposeIngress }}
hostnames:
- {{ printf "observability-proxy.%s" $host | quote }}
rules:
- backendRefs:
- name: observability-proxy
port: 80
{{- /* Without a request timeout Cilium leaves Envoy's 15s route default
in place, which would cut every tail. The proxy bounds each
request itself, so the route disables it. Envoy's 300s stream
idle timeout still applies; the proxy's tail idle limit stays
below it (see deployment.yaml). */}}
timeouts:
request: 0s
{{- end }}
@@ -0,0 +1,53 @@
{{- $cluster := .Values._cluster | default dict }}
{{- $solver := (index $cluster "solver") | default "http01" }}
{{- $clusterIssuer := (index $cluster "issuer-name") | default "letsencrypt-prod" }}
{{- $host := index $cluster "root-host" }}
{{- $exposeServices := splitList "," ((index $cluster "expose-services") | default "") }}
{{- $exposeIngress := (index $cluster "expose-ingress") | default "tenant-root" }}
{{- $gatewayEnabled := (index $cluster "gateway-enabled") | default "false" }}
{{- $wildcardSecret := (index $cluster "wildcard-secret-name") | default "" }}
{{- if and (has "observability-proxy" $exposeServices) (ne $gatewayEnabled "true") }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: observability-proxy
annotations:
{{- if not $wildcardSecret }}
cert-manager.io/cluster-issuer: {{ $clusterIssuer }}
{{- if eq $solver "http01" }}
acme.cert-manager.io/http01-ingress-ingressclassname: {{ $exposeIngress }}
{{- end }}
{{- end }}
{{- /* The proxy refuses a body above 1 MiB itself; nginx's 1m is the
same bound, so an oversized body stops at the edge. */}}
nginx.ingress.kubernetes.io/proxy-body-size: 1m
{{- /* A tail flushes every line; buffered, lines would reach the client
in batches. */}}
nginx.ingress.kubernetes.io/proxy-buffering: "off"
{{- /* The controller default is 10s. The longest silence the proxy
allows is a tail with no new line for --tail-idle-timeout (4m30s),
after which it ends the response itself; before the first byte
the token and access reviews may add up to --request-timeout (1m). */}}
nginx.ingress.kubernetes.io/proxy-read-timeout: "360"
spec:
ingressClassName: {{ $exposeIngress }}
rules:
- host: observability-proxy.{{ $host }}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: observability-proxy
port:
name: http
tls:
- hosts:
- observability-proxy.{{ $host }}
{{- /* Wildcard mode: ingress-nginx serves its default certificate, and
the tls section stays to keep the HTTP-to-HTTPS redirect. */}}
{{- if not $wildcardSecret }}
secretName: observability-proxy-tls
{{- end }}
{{- end }}
@@ -40,6 +40,15 @@ tests:
path: spec.template.spec.containers[0].args
content: --auth-cache-ttl=10s
# Envoy behind a Cilium Gateway resets a stream after 300s without data
# (envoy.streamIdleTimeoutDurationSeconds), so a silent tail has to be
# ended by the proxy first, or the client sees a reset instead of an end.
- it: ends a silent tail before the Gateway stream idle timeout
asserts:
- contains:
path: spec.template.spec.containers[0].args
content: --tail-idle-timeout=4m30s
- it: passes the metrics store names and short-term retention
asserts:
- contains:
@@ -0,0 +1,196 @@
suite: observability-proxy public endpoint
# Clients outside the cluster reach the proxy on its own host with a
# Kubernetes bearer token, so the route must pass Authorization and the
# /apis/... path through untouched, and must not end a request before the
# proxy does: a non-tail request may take up to --request-timeout (1m), and
# a tail may stay silent up to --tail-idle-timeout (4m30s). Whole objects are
# compared rather than single keys, so an annotation or a field that should
# not be there fails the test too.
templates:
- templates/ingress.yaml
- templates/httproute.yaml
release:
name: observability-proxy
namespace: cozy-observability-proxy
tests:
- it: publishes nothing without the cluster values
asserts:
- template: templates/ingress.yaml
hasDocuments:
count: 0
- template: templates/httproute.yaml
hasDocuments:
count: 0
- it: publishes nothing while observability-proxy is not in expose-services
set:
_cluster:
root-host: example.org
expose-services: "api,dashboard"
gateway-enabled: "false"
asserts:
- template: templates/ingress.yaml
hasDocuments:
count: 0
- template: templates/httproute.yaml
hasDocuments:
count: 0
- it: publishes nothing in Gateway mode while observability-proxy is not in expose-services
set:
_cluster:
root-host: example.org
expose-services: "api,dashboard"
gateway-enabled: "true"
asserts:
- template: templates/ingress.yaml
hasDocuments:
count: 0
- template: templates/httproute.yaml
hasDocuments:
count: 0
- it: renders an Ingress with a per-host certificate when gateway is off
set:
_cluster:
root-host: example.org
expose-services: "dashboard,observability-proxy"
expose-ingress: tenant-public
gateway-enabled: "false"
asserts:
- template: templates/httproute.yaml
hasDocuments:
count: 0
- template: templates/ingress.yaml
isKind:
of: Ingress
- template: templates/ingress.yaml
equal:
path: metadata.name
value: observability-proxy
- template: templates/ingress.yaml
equal:
path: metadata.annotations
value:
cert-manager.io/cluster-issuer: letsencrypt-prod
acme.cert-manager.io/http01-ingress-ingressclassname: tenant-public
nginx.ingress.kubernetes.io/proxy-body-size: 1m
nginx.ingress.kubernetes.io/proxy-buffering: "off"
nginx.ingress.kubernetes.io/proxy-read-timeout: "360"
- template: templates/ingress.yaml
equal:
path: spec
value:
ingressClassName: tenant-public
rules:
- host: observability-proxy.example.org
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: observability-proxy
port:
name: http
tls:
- hosts:
- observability-proxy.example.org
secretName: observability-proxy-tls
- it: leaves the http01 ingress class out under the dns01 solver
set:
_cluster:
root-host: example.org
expose-services: "observability-proxy"
gateway-enabled: "false"
solver: dns01
issuer-name: letsencrypt-stage
asserts:
- template: templates/ingress.yaml
equal:
path: metadata.annotations
value:
cert-manager.io/cluster-issuer: letsencrypt-stage
nginx.ingress.kubernetes.io/proxy-body-size: 1m
nginx.ingress.kubernetes.io/proxy-buffering: "off"
nginx.ingress.kubernetes.io/proxy-read-timeout: "360"
- template: templates/ingress.yaml
equal:
path: spec.ingressClassName
value: tenant-root
- it: relies on the wildcard certificate and keeps the TLS section for the redirect
set:
_cluster:
root-host: example.org
expose-services: "observability-proxy"
gateway-enabled: "false"
wildcard-secret-name: wildcard-tls
asserts:
- template: templates/ingress.yaml
equal:
path: metadata.annotations
value:
nginx.ingress.kubernetes.io/proxy-body-size: 1m
nginx.ingress.kubernetes.io/proxy-buffering: "off"
nginx.ingress.kubernetes.io/proxy-read-timeout: "360"
- template: templates/ingress.yaml
equal:
path: spec.tls
value:
- hosts:
- observability-proxy.example.org
- it: renders an HTTPRoute without a route timeout when gateway is on
set:
_cluster:
root-host: example.org
expose-services: "observability-proxy"
expose-ingress: tenant-public
gateway-enabled: "true"
asserts:
- template: templates/ingress.yaml
hasDocuments:
count: 0
- template: templates/httproute.yaml
isKind:
of: HTTPRoute
- template: templates/httproute.yaml
isAPIVersion:
of: gateway.networking.k8s.io/v1
- template: templates/httproute.yaml
equal:
path: metadata.name
value: observability-proxy
- template: templates/httproute.yaml
equal:
path: spec
value:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: cozystack
namespace: tenant-public
hostnames:
- observability-proxy.example.org
rules:
- backendRefs:
- name: observability-proxy
port: 80
timeouts:
request: 0s
- it: attaches to the tenant-root Gateway when expose-ingress is unset
set:
_cluster:
root-host: example.org
expose-services: "observability-proxy"
gateway-enabled: "true"
asserts:
- template: templates/httproute.yaml
equal:
path: spec.parentRefs[?(@.name == "cozystack")].namespace
value: tenant-root