<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Adds an optional public endpoint for the observability proxy, so that clients outside the cluster (kubectl, scripts, third-party tools) can call it directly with their Kubernetes bearer token. The existing paths do not deliver the token: kube-apiserver removes the `Authorization` header after it authenticates a request, so the service proxy of kube-apiserver cannot carry it, and the console route goes through the dashboard gatekeeper, which with OIDC enabled answers a client without a session cookie with 401 and exists only where the dashboard is published. The endpoint is off by default. It is published at `observability-proxy.<publishing.host>` once `observability-proxy` is added to `publishing.exposedServices`, in the same two modes the dashboard uses: - `gateway.enabled: false`: an Ingress on the publishing ingress class, with a per-host cert-manager certificate, or the operator's wildcard certificate when `publishing.certificates.wildcardSecretName` is set (the TLS section stays, so HTTP is still redirected to HTTPS). - `gateway.enabled: true`: an HTTPRoute on the publishing tenant's Gateway. The platform now lists `cozy-observability-proxy` in `gateway.attachedNamespaces` so that the Gateway admits the route. A cluster that overrides `gateway.attachedNamespaces` replaces the whole list and has to add the namespace itself. In Ingress mode the request crosses the publishing tenant's egress policies, which deny everything except the `allow-to-*` destinations, so the tenant chart gains `allow-to-observability-proxy`. It matches the whole `cozy-observability-proxy` namespace, as `allow-to-linstor-gui` does, because cert-manager runs the HTTP-01 solver pods for the proxy's certificate there. The route passes the request through unchanged: no path rewrite and no authentication at the edge, so `/apis/...` and the bearer token reach the proxy as sent. The edge limits are set so that they never end a request before the proxy does: - The tenant ingress-nginx reads upstream with a 10s timeout by default, while the proxy allows a non-tail request up to `--request-timeout` (1m) and keeps a log tail open until it has been silent for `--tail-idle-timeout`. Before the first byte of a response, the token and access reviews can add up to another minute. The Ingress sets a 360s read timeout and keeps response buffering off, so tail lines reach the client as they arrive. - The Ingress sets a 1 MiB body limit, the same limit the proxy enforces on a request body. - On an HTTPRoute that sets no timeout, Cilium keeps Envoy's 15s route timeout, which would cut every tail. The route disables it (`timeouts.request: 0s`), because the proxy limits each request itself. Envoy's stream idle timeout (Cilium `envoy.streamIdleTimeoutDurationSeconds`, 300s by default) still applies and cannot be changed from an HTTPRoute, so the chart now runs the proxy with `--tail-idle-timeout=4m30s` instead of the 5m default. A tail that stops sending lines is then ended by the proxy as a complete response, not reset by Envoy. Before the first byte, slow token and access reviews together with a log store that never sends headers can still exceed 300s; that case ends in an error either way. The proxy never reads the client address or `X-Forwarded-*`. All its limits and its logs use the authenticated user, so nothing changes behind an ingress and the client cannot spoof anything through headers. ### Security This adds a new internet-facing entry point when it is enabled. What protects it: - Every request to a tenant root needs a bearer token that kube-apiserver accepts through a TokenReview, and a SubjectAccessReview for the requested namespace. Paths the proxy does not serve return 404 before any review, and a request without a bearer token returns 401 without a review. - Per-user limits apply after authentication: request rate, concurrent requests, concurrent tails per user, and a total cap on open tails. Every request is bounded by the request timeout, and every tail by the idle timeout. - The body is capped at 1 MiB at the edge and in the proxy. - The new tenant egress policy lets every pod of a tenant open a connection to the proxy namespace, whether or not the proxy is published. This gives a tenant workload nothing it cannot already do through the published host, and the proxy asks for a token in both cases. What it does not protect against: an unauthenticated client can make the proxy send one TokenReview per distinct token it invents. Rejected tokens are cached by hash for the auth cache TTL, so a repeated token costs nothing more. This is the same cost a client can already cause by calling the published Kubernetes API (`api` is in `publishing.exposedServices` by default). No per-IP rate limit is added at the edge, because with `publishing.externalIPs` the ingress Service uses `externalTrafficPolicy: Cluster`, which can hide client addresses behind node addresses, so a per-IP limit would throttle all clients together. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> Not a UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#787 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff changes `packages/core/platform/values.yaml`, which reaches cozystack/website: `content/en/docs/next/operations/configuration/platform-package.md` lists the possible values of `publishing.exposedServices` and the default `gateway.attachedNamespaces` list, and both need the new entries. The observability-proxy documentation is tracked in cozystack/website#787, which does not cover the public endpoint yet; the endpoint and the new list entries will be added there once this PR is accepted. No other repository in the trigger map is reached by this diff. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(observability-proxy): the observability proxy can be published at observability-proxy.<host> for clients outside the cluster that use a Kubernetes bearer token. It is off by default; add `observability-proxy` to `publishing.exposedServices` to publish it, through an Ingress or, with `gateway.enabled`, an HTTPRoute. Clusters that override `gateway.attachedNamespaces` must add `cozy-observability-proxy` to it for the Gateway mode. Tenants gain an egress policy to the proxy namespace, which the Ingress mode needs. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Tenant workloads can now connect to the observability proxy. * The observability proxy can be published through an Ingress or Gateway route when enabled in the service exposure settings. * Proxy routing supports long-running requests and streams with adjusted timeout handling. * Ingress publishing supports per-host or wildcard certificates and configurable certificate challenge settings. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Tenant
A tenant is the main unit of security on the platform. The closest analogy would be Linux kernel namespaces.
Tenants can be created recursively and are subject to the following rules:
Tenant naming
Tenant names must be alphanumeric:
- Lowercase letters (
a-z) and digits (0-9) only - Must start with a lowercase letter
- Dashes (
-) are not allowed, unlike with other services - Maximum length depends on the cluster configuration (Helm release prefix and root domain)
This restriction exists to keep consistent naming in tenants, nested tenants, and services deployed in them.
A tenant cannot be named foo-bar because parsing internal resource names like tenant-foo-bar would be ambiguous.
For example:
- The root tenant is named
root, but internally it's referenced astenant-root. - A nested tenant could be named
foo, which would result intenant-fooin service names and URLs.
Unique domains
Each tenant has its own domain.
By default, (unless otherwise specified), it inherits the domain of its parent with a prefix of its name.
For example, if the parent had the domain example.org, then tenant-foo would get the domain foo.example.org by default.
Kubernetes clusters created in this tenant namespace would get domains like: kubernetes-cluster.foo.example.org
Example:
tenant-root (example.org)
└── tenant-foo (foo.example.org)
└── kubernetes-cluster1 (kubernetes-cluster1.foo.example.org)
Nesting tenants and reusing parent services
Tenants can be nested. A tenant administrator can create nested tenants using the "Tenant" application from the catalogue.
Higher-level tenants can view and manage the applications of all their children tenants. If a tenant does not run their own cluster services, it can access ones of its parent.
For example, you create:
- Tenant
tenant-u1with a set of services likeetcd,ingress,monitoring. - Tenant
tenant-u2nested intenant-u1.
Let's see what will happen when you run Kubernetes and Postgres under tenant-u2 namespace.
Since tenant-u2 does not have its own cluster services like etcd, ingress, and monitoring,
the applications running in tenant-u2 will use the cluster services of the parent tenant.
This in turn means:
- The Kubernetes cluster data will be stored in
etcdfortenant-u1. - Access to the cluster will be through the common
ingressoftenant-u1. - Essentially, all metrics will be collected in the
monitoringfromtenant-u1, and only that tenant will have access to them.
Example:
tenant-u1
├── etcd
├── ingress
├── monitoring
└── tenant-u2
├── kubernetes-cluster1
└── postgres-db1
Network isolation
Tenant network isolation is enforced with Cilium network policies. On the isp-hosted and isp-hosted-slim platform variants the CNI belongs to the host, so these policies are created only if the host serves the Cilium policy API when the tenant is installed or updated. On a host without Cilium, tenants get no network isolation from Cozystack, and isolating them is up to the host. If Cilium appears on the host later, the policies are created on the tenant's next chart or values change, not automatically.
Parameters
Common parameters
| Name | Description | Type | Value |
|---|---|---|---|
host |
The hostname used to access tenant services (defaults to using the tenant name as a subdomain for its parent tenant host). | string |
"" |
etcd |
Deploy own Etcd cluster. | bool |
false |
monitoring |
Deploy own Monitoring Stack. | bool |
false |
tracingCentral |
Send this tenant's traces to the shared VictoriaTraces store in tenant-root instead of a per-tenant backend; its Grafana reads them through a vmauth there that pins the tenant's own account, so tenants sharing the store cannot read each other's traces. It needs the tenant's own Monitoring (monitoring: true) and tenant-root hosting the store (tracingCentralHost: true on its Monitoring, with a cluster-mode tracingStorages entry named generic); without that the tenant's monitoring render fails rather than dropping spans. tenant-root, and whoever it admits to its Grafana, are inside the store's trust boundary. Applications should export over OTLP/HTTP, in requests below 3 MiB: spans above the platform's per-tenant write rate are retried over HTTP but dropped over gRPC, and a larger request is refused and not retried. |
bool |
false |
ingress |
Deploy own Ingress Controller. | bool |
false |
gateway |
Deploy own Gateway API Gateway (backed by Cilium Gateway API controller). When unset (the default) or set to false, the tenant does not own a Gateway and instead publishes through the nearest ancestor tenant that owns one, regardless of whether host is empty or custom (false currently behaves the same as unset and does not opt out of that inheritance). Set to true to give the tenant its own Gateway with its own LoadBalancer IP and Certificate. A tenant with a custom (non-derived) host that needs its apex publicly reachable must set true explicitly, because an ancestor's TLS certificate does not cover a custom apex. Note: leave the key absent (do not write gateway: null): the chart distinguishes "unset" via missing-key, not via null value, to satisfy the JSON schema generated from this comment. |
bool |
false |
gatewayClass |
GatewayClass for this tenant's own Gateway. Empty uses the platform default (gateway.className). Only the classes the operator lists in the platform's gateway.tenantSelectableClasses, plus that default, may be named here — anything else fails this tenant's own gateway release and reaches no other tenant. Read only when this tenant owns a Gateway; an inheriting tenant uses its ancestor's. Whether a class terminates TLS itself or at its provider's edge is declared once per class on the platform, not here. |
string |
"" |
seaweedfs |
Deploy own SeaweedFS. | bool |
false |
computeplane |
Deploy own ComputePlane — a single-tenant, Cozystack-managed cluster for untrusted-code applications. The tenant receives no admin kubeconfig for it. Automatic routing of catalog applications onto it (placement: ComputePlane) is a planned follow-up and is not available yet; until it lands, external catalogs target the cluster via its computeplane-cluster-admin-kubeconfig Secret. See design-proposals/compute-plane in cozystack/community. | bool |
false |
schedulingClass |
The name of a SchedulingClass CR to apply scheduling constraints for this tenant's workloads. | string |
"" |
resourceQuotas |
Define resource quotas for the tenant. | map[string]quantity |
{} |
Configuration
Resource Quotas
The resourceQuotas parameter allows you to limit resources available to the tenant. Supported keys include:
Compute resources (converted to requests.X and limits.X):
cpu- Total CPU cores (e.g.,"4"or"500m")memory- Total memory (e.g.,"4Gi"or"512Mi")ephemeral-storage- Ephemeral storage limit (e.g.,"10Gi")storage- Total persistent storage (e.g.,"100Gi")
Object count quotas (passed as-is):
pods- Maximum number of podsservices- Maximum number of servicesservices.loadbalancers- Maximum number of LoadBalancer servicesservices.nodeports- Maximum number of NodePort servicesconfigmaps- Maximum number of ConfigMapssecrets- Maximum number of Secretspersistentvolumeclaims- Maximum number of PVCs
Example:
resourceQuotas:
cpu: 4
memory: 4Gi
storage: 10Gi
services.loadbalancers: "3"
pods: "50"