<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Builds on #5219 and #5220, both merged. The namespace metrics root `/apis/cozystack.io/v1alpha1/namespaces/<ns>/metrics/...` now also returns series that a platform exporter attributes to `<ns>`. Example: the ClickHouse operator's exporter runs in `cozy-clickhouse-operator`. vmagent scrapes it without `honorLabels`, so its series carry `namespace="cozy-clickhouse-operator"`, and the tenant namespace the exporter reported moves to `exported_namespace="<ns>"`. Until now, any client of the namespace root (kubectl, automation, the console) never saw those series. ### Behavior changes - Namespace root, every endpoint (`query`, `query_range`, `series`, `labels`, `label/<name>/values`): - was: one proxy-set `extra_filters[]={namespace="<ns>"}`; - now: two values, `{namespace="<ns>"}` and `{namespace=~"cozy-.*",exported_namespace="<ns>"}`. vmselect ORs them. - Label APIs without a client `match[]`: the proxy already repeats the scope in `match[]` for `-search.ignoreExtraFiltersAtLabelsAPI`; it now repeats both values. vmselect ANDs every `match[]` with every `extra_filters[]`. Each cross pair is narrower than one of its halves, so the result is still exactly the union. - Client `extra_filters`, `extra_filters[]` and `extra_label` are still dropped. - New flag `--metrics-platform-namespace-prefix` (default `cozy-`) and chart value `observabilityProxy.metrics.platformNamespacePrefix`. An empty value turns the second branch off and restores the single-value scope. The proxy quotes the prefix and anchors it, so the value is matched literally and cannot become a wider regex. The flag accepts only namespace name characters, because any other value would match no namespace and silently drop the exporter series. - Exporter series keep `namespace="cozy-..."`. A query that also selects `namespace="<ns>"` excludes them; select them by `exported_namespace` or by the exporter's own labels (for ClickHouse, `chi`). - The branch is not specific to ClickHouse. For example, kube-state-metrics in the platform monitoring namespace sets `exported_namespace` from `metadata.namespace` on the Flux `gotk_resource_info` series (`packages/system/monitoring-agents/values.yaml`), so a namespace root now also returns the state of that namespace's own HelmReleases and Kustomizations. This is the namespace's own data, but it is a visible change with the default prefix. - Queries grouped `by (namespace)` can now return an extra group labelled with the exporter's platform namespace. The exporter's series also carry its own target labels (`pod`, `instance`, `job`) into the tenant's results. - An empty scope is refused with 500 instead of forwarding a request without `extra_filters[]`. - The application roots of #5220 are not affected: they replace the scope with their own selectors. ### Threat model 1. **Scrape path.** vmagent sets `namespace` from service discovery, while `exported_namespace` is whatever the exporter wrote. Any tenant pod can write a `namespace` label, which arrives as `exported_namespace`, so the extra branch only trusts it on series scraped from platform namespaces. Tenant roles (`packages/system/cozystack-basics/templates/clusterroles.yaml`) grant no access to `operator.victoriametrics.com` and no verbs on `Namespace`. A tenant can therefore neither run a target in `cozy-*` nor point a scrape object at one. 2. **Remote-write writers.** A tenant's nested Kubernetes cluster runs its own vmagent, which remote-writes into the same store when the tenant has no monitoring of its own, and the labels it writes are not enforced. Inside that cluster the tenant can create `cozy-*` namespaces or any other, so it can already write `namespace="<any ns>"` and pass the existing single filter. This is an existing limitation that this PR neither introduces nor widens: anything that can write `namespace="cozy-x",exported_namespace="<ns>"` can write `namespace="<ns>"` directly. Closing it needs enforcement on the write path and is out of scope here. 3. **Impact of a forged series.** The scope only adds series to a namespace's results; it never lets a caller read another namespace. A forged series can spoil or flood another namespace's graphs, up to the series limits that answer 422. ### Rejected alternatives - **Bare `{exported_namespace="<ns>"}`.** Every namespace is scraped by default, so any tenant pod could inject series into another namespace's root. - **Relabeling the ClickHouse operator's scrape so that `namespace` carries the tenant namespace.** It fixes one exporter at a time, each in its own chart, and it changes the label set of every existing series, which splits their history at the change and breaks any query that selects them by the operator's namespace. The proxy-side branch covers every platform exporter and leaves stored series as they are. - **Scraping the ClickHouse pods directly.** The metric names differ from the operator's exporter, so the shipped ClickHouse dashboard (`dashboards/clickhouse/altinity-clickhouse-operator-dashboard.json`), which reads the exporter's `chi_clickhouse_*` series, would have to be rewritten. - **A free regex flag instead of a prefix.** One typo (`.*`) would trust `exported_namespace` from every namespace. ### Tests - Table of namespace-root requests: exactly two `extra_filters[]` values in order with the default prefix, one value with an empty prefix, on all five endpoints. Client `extra_filters[]` still dropped. - A model of vmselect's selection (OR over `extra_filters[]`, AND with each `match[]`) shows that `namespace=cozy-x, exported_namespace=<ns>` is visible, and that `namespace=tenant-y, exported_namespace=<ns>` and `namespace=xcozy-a, ...` are not. - Escaping of `<ns>` in `exported_namespace`, literal prefix quoting, empty scope refused, flag validation, and a chart test that an empty value reaches the binary as an explicit empty flag. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> Not a UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches only `cmd/observability-proxy`, `internal/observabilityproxy` and the system package `packages/system/observability-proxy` (a new value and flag, no app, platform values, CRD, `hack/` or naming change). None of the triggers in `docs/agents/contributing.md` under "Downstream Repositories" apply. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(observability-proxy): the namespace metrics root also returns series that exporters in platform (`cozy-*`) namespaces attribute to the namespace through `exported_namespace`, such as the ClickHouse operator's. The prefix is set by `--metrics-platform-namespace-prefix` / `observabilityProxy.metrics.platformNamespacePrefix`; an empty value disables it. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Metrics queries now include series attributed to a tenant namespace by exporters in platform namespaces with the configured prefix. The default prefix is `cozy-`; setting it to an empty value disables this behavior. * Namespace scoping is applied across supported metrics endpoints, including label lookups. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Cozystack
Cozystack is a free and open-source platform and framework for building clouds.
Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack to build your own platform
You can install only the Cozystack components you need on an existing Kubernetes cluster
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.
Project documents
| Document | What it covers |
|---|---|
| Governance | How decisions are made and who makes them |
| Contributor Ladder | How to grow from contributor to maintainer |
| Maintainers | Who maintains the project |
| Code of Conduct | What we expect of each other |
| Security Policy | Reporting a vulnerability, and which releases get fixes |
| Adopters | Who runs Cozystack in production |
| Roadmap | Where the project is going |
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.
