feat(observability-proxy): include platform exporter series in the namespace root (#5277)

<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Builds on #5219 and #5220, both merged.

The namespace metrics root
`/apis/cozystack.io/v1alpha1/namespaces/<ns>/metrics/...` now also
returns series that a platform exporter attributes to `<ns>`. Example:
the ClickHouse operator's exporter runs in `cozy-clickhouse-operator`.
vmagent scrapes it without `honorLabels`, so its series carry
`namespace="cozy-clickhouse-operator"`, and the tenant namespace the
exporter reported moves to `exported_namespace="<ns>"`. Until now, any
client of the namespace root (kubectl, automation, the console) never
saw those series.

### Behavior changes

- Namespace root, every endpoint (`query`, `query_range`, `series`,
`labels`, `label/<name>/values`):
  - was: one proxy-set `extra_filters[]={namespace="<ns>"}`;
- now: two values, `{namespace="<ns>"}` and
`{namespace=~"cozy-.*",exported_namespace="<ns>"}`. vmselect ORs them.
- Label APIs without a client `match[]`: the proxy already repeats the
scope in `match[]` for `-search.ignoreExtraFiltersAtLabelsAPI`; it now
repeats both values. vmselect ANDs every `match[]` with every
`extra_filters[]`. Each cross pair is narrower than one of its halves,
so the result is still exactly the union.
- Client `extra_filters`, `extra_filters[]` and `extra_label` are still
dropped.
- New flag `--metrics-platform-namespace-prefix` (default `cozy-`) and
chart value `observabilityProxy.metrics.platformNamespacePrefix`. An
empty value turns the second branch off and restores the single-value
scope. The proxy quotes the prefix and anchors it, so the value is
matched literally and cannot become a wider regex. The flag accepts only
namespace name characters, because any other value would match no
namespace and silently drop the exporter series.
- Exporter series keep `namespace="cozy-..."`. A query that also selects
`namespace="<ns>"` excludes them; select them by `exported_namespace` or
by the exporter's own labels (for ClickHouse, `chi`).
- The branch is not specific to ClickHouse. For example,
kube-state-metrics in the platform monitoring namespace sets
`exported_namespace` from `metadata.namespace` on the Flux
`gotk_resource_info` series
(`packages/system/monitoring-agents/values.yaml`), so a namespace root
now also returns the state of that namespace's own HelmReleases and
Kustomizations. This is the namespace's own data, but it is a visible
change with the default prefix.
- Queries grouped `by (namespace)` can now return an extra group
labelled with the exporter's platform namespace. The exporter's series
also carry its own target labels (`pod`, `instance`, `job`) into the
tenant's results.
- An empty scope is refused with 500 instead of forwarding a request
without `extra_filters[]`.
- The application roots of #5220 are not affected: they replace the
scope with their own selectors.

### Threat model

1. **Scrape path.** vmagent sets `namespace` from service discovery,
while `exported_namespace` is whatever the exporter wrote. Any tenant
pod can write a `namespace` label, which arrives as
`exported_namespace`, so the extra branch only trusts it on series
scraped from platform namespaces. Tenant roles
(`packages/system/cozystack-basics/templates/clusterroles.yaml`) grant
no access to `operator.victoriametrics.com` and no verbs on `Namespace`.
A tenant can therefore neither run a target in `cozy-*` nor point a
scrape object at one.
2. **Remote-write writers.** A tenant's nested Kubernetes cluster runs
its own vmagent, which remote-writes into the same store when the tenant
has no monitoring of its own, and the labels it writes are not enforced.
Inside that cluster the tenant can create `cozy-*` namespaces or any
other, so it can already write `namespace="<any ns>"` and pass the
existing single filter. This is an existing limitation that this PR
neither introduces nor widens: anything that can write
`namespace="cozy-x",exported_namespace="<ns>"` can write
`namespace="<ns>"` directly. Closing it needs enforcement on the write
path and is out of scope here.
3. **Impact of a forged series.** The scope only adds series to a
namespace's results; it never lets a caller read another namespace. A
forged series can spoil or flood another namespace's graphs, up to the
series limits that answer 422.

### Rejected alternatives

- **Bare `{exported_namespace="<ns>"}`.** Every namespace is scraped by
default, so any tenant pod could inject series into another namespace's
root.
- **Relabeling the ClickHouse operator's scrape so that `namespace`
carries the tenant namespace.** It fixes one exporter at a time, each in
its own chart, and it changes the label set of every existing series,
which splits their history at the change and breaks any query that
selects them by the operator's namespace. The proxy-side branch covers
every platform exporter and leaves stored series as they are.
- **Scraping the ClickHouse pods directly.** The metric names differ
from the operator's exporter, so the shipped ClickHouse dashboard
(`dashboards/clickhouse/altinity-clickhouse-operator-dashboard.json`),
which reads the exporter's `chi_clickhouse_*` series, would have to be
rewritten.
- **A free regex flag instead of a prefix.** One typo (`.*`) would trust
`exported_namespace` from every namespace.

### Tests

- Table of namespace-root requests: exactly two `extra_filters[]` values
in order with the default prefix, one value with an empty prefix, on all
five endpoints. Client `extra_filters[]` still dropped.
- A model of vmselect's selection (OR over `extra_filters[]`, AND with
each `match[]`) shows that `namespace=cozy-x, exported_namespace=<ns>`
is visible, and that `namespace=tenant-y, exported_namespace=<ns>` and
`namespace=xcozy-a, ...` are not.
- Escaping of `<ns>` in `exported_namespace`, literal prefix quoting,
empty scope refused, flag validation, and a chart test that an empty
value reaches the binary as an explicit empty flag.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not a UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `cmd/observability-proxy`,
`internal/observabilityproxy` and the system package
`packages/system/observability-proxy` (a new value and flag, no app,
platform values, CRD, `hack/` or naming change). None of the triggers in
`docs/agents/contributing.md` under "Downstream Repositories" apply.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(observability-proxy): the namespace metrics root also returns series that exporters in platform (`cozy-*`) namespaces attribute to the namespace through `exported_namespace`, such as the ClickHouse operator's. The prefix is set by `--metrics-platform-namespace-prefix` / `observabilityProxy.metrics.platformNamespacePrefix`; an empty value disables it.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Metrics queries now include series attributed to a tenant namespace by
exporters in platform namespaces with the configured prefix. The default
prefix is `cozy-`; setting it to an empty value disables this behavior.
* Namespace scoping is applied across supported metrics endpoints,
including label lookups.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
Aleksei Sviridkin authored and GitHub committed 2026-11-14 02:44:48 +03:00
commit 3ede7c4d33
12 files changed
+341 -21

No files matched your search

+6
View File
@@ -101,6 +101,7 @@ func parseFlags(args []string) (options, error) {
fs.DurationVar(&o.metrics.App.Lookback, "metrics-app-lookback", 30*24*time.Hour, "how far before the window an application metrics query may read; one reading further back, or by an unknown amount, answers 422")
fs.IntVar(&o.metrics.App.MaxPods, "metrics-app-max-pods", 500, "most pods of one application in the lookup window; more answer 422")
fs.IntVar(&o.metrics.App.MaxPVCs, "metrics-app-max-pvcs", 500, "most PVCs of one application in the lookup window; more answer 422")
fs.StringVar(&o.metrics.PlatformNamespacePrefix, "metrics-platform-namespace-prefix", "cozy-", "namespaces with this prefix run platform exporters, whose exported_namespace label adds series to that namespace's scope; empty disables")
fs.IntVar(&o.limiter.MaxConcurrentPerUser, "max-concurrent-per-user", 10, "concurrent non-tail requests per user; 0 disables")
fs.Float64Var(&o.limiter.RatePerUser, "rate-limit-per-user", 10, "sustained client requests per second per user; an application metrics request makes three store calls (pod lookup, PVC lookup, query); 0 disables")
fs.IntVar(&o.limiter.RateBurst, "rate-limit-burst", 20, "request burst per user")
@@ -121,6 +122,11 @@ func parseFlags(args []string) (options, error) {
return o, fmt.Errorf("invalid metrics store name %q", store)
}
}
// The prefix is matched literally, so one no namespace name can start
// with would turn the exporter branch off without a word.
if p := o.metrics.PlatformNamespacePrefix; p != "" && len(validation.IsDNS1123Label(p+"x")) > 0 {
return o, fmt.Errorf("invalid metrics platform namespace prefix %q", p)
}
switch {
case o.rootTenant == "":
return o, errors.New("--root-tenant must not be empty")
+16 -1
View File
@@ -46,6 +46,10 @@ func TestParseFlagsDefaults(t *testing.T) {
if m.RootTenant != o.rootTenant {
t.Errorf("metrics root tenant = %q, want --root-tenant", m.RootTenant)
}
// The prefix every platform namespace carries.
if m.PlatformNamespacePrefix != "cozy-" {
t.Errorf("platform namespace prefix = %q, want cozy-", m.PlatformNamespacePrefix)
}
}
func TestParseFlagsOverrides(t *testing.T) {
@@ -67,6 +71,7 @@ func TestParseFlagsMetricsOverrides(t *testing.T) {
"--root-tenant=tenant-x", "--metrics-shortterm-store=fast", "--metrics-longterm-store=slow",
"--metrics-shortterm-retention=24h", "--metrics-max-range=168h", "--metrics-max-points=500", "--metrics-max-timeout=10s",
"--metrics-app-lookback=2h", "--metrics-app-max-pods=7", "--metrics-app-max-pvcs=3",
"--metrics-platform-namespace-prefix=sys-",
})
if err != nil {
t.Fatal(err)
@@ -74,9 +79,15 @@ func TestParseFlagsMetricsOverrides(t *testing.T) {
m := o.metrics
if m.RootTenant != "tenant-x" || m.ShortTermStore != "fast" || m.LongTermStore != "slow" || m.ShortTermRetention != 24*time.Hour ||
m.Limits.MaxRange != 168*time.Hour || m.Limits.MaxPoints != 500 || m.Limits.MaxTimeout != 10*time.Second ||
m.App.Lookback != 2*time.Hour || m.App.MaxPods != 7 || m.App.MaxPVCs != 3 {
m.App.Lookback != 2*time.Hour || m.App.MaxPods != 7 || m.App.MaxPVCs != 3 ||
m.PlatformNamespacePrefix != "sys-" {
t.Errorf("overrides not applied: %+v", m)
}
// An empty prefix switches the exporter branch of the scope off.
o, err = parseFlags([]string{"--metrics-platform-namespace-prefix="})
if err != nil || o.metrics.PlatformNamespacePrefix != "" {
t.Errorf("empty platform namespace prefix: %q, %v", o.metrics.PlatformNamespacePrefix, err)
}
}
func TestParseFlagsInvalid(t *testing.T) {
@@ -112,6 +123,10 @@ func TestParseFlagsInvalid(t *testing.T) {
{"--metrics-app-lookback=0s"},
{"--metrics-app-max-pods=0"},
{"--metrics-app-max-pvcs=0"},
// No namespace name starts with these, so they would match nothing.
{"--metrics-platform-namespace-prefix=Cozy-"},
{"--metrics-platform-namespace-prefix=cozy.*"},
{"--metrics-platform-namespace-prefix=-"},
{"--unknown"},
// The proxy would cut the request while vmselect is still within
// the timeout it was given.
+37 -8
View File
@@ -22,6 +22,7 @@ import (
"net/http"
"net/url"
"regexp"
"slices"
"strconv"
"strings"
"time"
@@ -47,6 +48,9 @@ type MetricsConfig struct {
ShortTermRetention time.Duration
Limits MetricsLimits
App AppScopeLimits
// PlatformNamespacePrefix names the namespaces whose exporters may
// attribute a series to another namespace; empty trusts none.
PlatformNamespacePrefix string
// UpstreamURL maps a store name to its Prometheus API base URL, without
// a trailing slash; nil means the in-cluster vmselect Service.
UpstreamURL func(store string) string
@@ -103,15 +107,37 @@ func namespaceSelector(ns string) string {
return "{namespace=" + strconv.Quote(ns) + "}"
}
// namespaceScope selects the series of namespace ns. Besides its own, it
// takes the series an exporter in a platform namespace attributes to ns:
// vmagent sets namespace to the scraped target's and, without honorLabels,
// moves the exporter's own namespace label to exported_namespace. Any pod
// can write that label, so it is trusted only where the platform's vmagent
// set namespace to one with the platform prefix, in which tenants cannot run
// pods or scrape objects. Series remote-written from a tenant's nested
// cluster carry whatever namespace that cluster chose, so they are outside
// this guarantee for both values alike.
func (c MetricsConfig) namespaceScope(ns string) []string {
scope := []string{namespaceSelector(ns)}
if c.PlatformNamespacePrefix != "" {
// MetricsQL anchors =~ at both ends.
platform := strconv.Quote(regexp.QuoteMeta(c.PlatformNamespacePrefix) + ".*")
scope = append(scope, "{namespace=~"+platform+",exported_namespace="+strconv.Quote(ns)+"}")
}
return scope
}
// BuildMetricsParams rebuilds the upstream parameters from an allowlist and
// adds the namespace scope as the only extra_filters[] value. vmselect ORs
// repeated extra_filters, so a client value would widen the scope; it ANDs
// the single one into every selector of the query and every match[], which
// is what lets the query pass through unparsed.
func BuildMetricsParams(endpoint string, in url.Values, namespace string, now time.Time, lim MetricsLimits) (metricsRequest, *HTTPError) {
// sends scope as the only extra_filters[] values. vmselect ORs repeated
// extra_filters, so a client value would widen the scope; it ANDs each into
// every selector of the query and every match[], which is what lets the
// query pass through unparsed.
func BuildMetricsParams(endpoint string, in url.Values, scope []string, now time.Time, lim MetricsLimits) (metricsRequest, *HTTPError) {
req := metricsRequest{path: endpoint, params: url.Values{}}
if len(scope) == 0 {
return req, errorf(http.StatusInternalServerError, "no metrics scope")
}
out := req.params
out.Set("extra_filters[]", namespaceSelector(namespace))
out["extra_filters[]"] = slices.Clone(scope)
timeout := lim.MaxTimeout
if v := in.Get("timeout"); v != "" {
d, herr := parseMetricsDuration("timeout", v)
@@ -226,8 +252,11 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti
}
// vmselect started with -search.ignoreExtraFiltersAtLabelsAPI drops
// extra_filters[] on the label APIs unless a match[] is present
// (getCommonParamsInternal), so the scope is repeated here.
out.Set("match[]", namespaceSelector(namespace))
// (getCommonParamsInternal), so the scope is repeated here. Where both
// are sent, vmselect ANDs every match[] with every extra_filters[];
// a cross pair is narrower than one of its halves, so the union is
// still the scope.
out["match[]"] = slices.Clone(scope)
req.scopedMatch = true
}
return req, nil
@@ -19,6 +19,7 @@ package observabilityproxy
import (
"net/http"
"net/url"
"slices"
"strconv"
"strings"
"testing"
@@ -29,13 +30,20 @@ func testMetricsLimits() MetricsLimits {
return MetricsLimits{MaxRange: 30 * 24 * time.Hour, MaxPoints: 11000, MaxTimeout: 30 * time.Second}
}
const wantMetricsScope = `{namespace="tenant-a"}`
// wantMetricsScope is the namespace root's scope under the default
// platform prefix: the namespace's own series, then the series a platform
// exporter attributes to it.
var wantMetricsScope = []string{`{namespace="tenant-a"}`, `{namespace=~"cozy-.*",exported_namespace="tenant-a"}`}
func testScope() []string {
return MetricsConfig{PlatformNamespacePrefix: "cozy-"}.namespaceScope("tenant-a")
}
func unix(t time.Time) string { return strconv.FormatInt(t.Unix(), 10) }
func buildMetrics(t *testing.T, endpoint string, in url.Values) metricsRequest {
t.Helper()
req, err := BuildMetricsParams(endpoint, in, "tenant-a", testNow, testMetricsLimits())
req, err := BuildMetricsParams(endpoint, in, testScope(), testNow, testMetricsLimits())
if err != nil {
t.Fatalf("BuildMetricsParams(%s, %v): %v", endpoint, in, err)
}
@@ -43,7 +51,7 @@ func buildMetrics(t *testing.T, endpoint string, in url.Values) metricsRequest {
}
func metricsStatus(endpoint string, in url.Values) int {
_, err := BuildMetricsParams(endpoint, in, "tenant-a", testNow, testMetricsLimits())
_, err := BuildMetricsParams(endpoint, in, testScope(), testNow, testMetricsLimits())
if err == nil {
return http.StatusOK
}
@@ -84,8 +92,8 @@ func TestBuildMetricsParamsAllowlist(t *testing.T) {
t.Errorf("parameter %q forwarded: %v", k, req.params)
}
}
if got := req.params["extra_filters[]"]; len(got) != 1 || got[0] != wantMetricsScope {
t.Errorf("extra_filters[] = %q, want exactly [%s]", got, wantMetricsScope)
if got := req.params["extra_filters[]"]; !slices.Equal(got, wantMetricsScope) {
t.Errorf("extra_filters[] = %q, want exactly %q", got, wantMetricsScope)
}
for _, m := range req.params["match[]"] {
if strings.Contains(m, "tenant-b") {
@@ -116,8 +124,8 @@ func TestNamespaceSelectorEscapes(t *testing.T) {
func TestBuildMetricsParamsLabelEndpointsAlwaysMatch(t *testing.T) {
for _, endpoint := range []string{"api/v1/labels", "api/v1/label/__name__/values"} {
req := buildMetrics(t, endpoint, url.Values{})
if got := req.params["match[]"]; len(got) != 1 || got[0] != wantMetricsScope {
t.Errorf("%s without match[] sends %q, want the namespace selector", endpoint, got)
if got := req.params["match[]"]; !slices.Equal(got, wantMetricsScope) {
t.Errorf("%s without match[] sends %q, want the namespace scope %q", endpoint, got, wantMetricsScope)
}
req = buildMetrics(t, endpoint, url.Values{"match[]": {"up", `{job="x"}`}})
if got := req.params["match[]"]; len(got) != 2 || got[0] != "up" || got[1] != `{job="x"}` {
@@ -198,7 +206,7 @@ func TestBuildMetricsParamsCanonicalForms(t *testing.T) {
})
want := url.Values{
"query": {"up"}, "start": {"2026-11-04T11:00:00.250Z"}, "end": {"2026-11-04T12:00:00.000Z"}, "step": {"90"},
"timeout": {"30"}, "extra_filters[]": {wantMetricsScope},
"timeout": {"30"}, "extra_filters[]": wantMetricsScope,
}
if req.params.Encode() != want.Encode() {
t.Errorf("params = %v\nwant %v", req.params, want)
@@ -0,0 +1,237 @@
/*
Copyright 2026 The Cozystack Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package observabilityproxy
import (
"net/http"
"net/url"
"regexp"
"slices"
"strconv"
"strings"
"testing"
"time"
)
func TestNamespaceScope(t *testing.T) {
tests := []struct {
prefix string
want []string
}{
{"cozy-", []string{`{namespace="tenant-a"}`, `{namespace=~"cozy-.*",exported_namespace="tenant-a"}`}},
{"", []string{`{namespace="tenant-a"}`}},
// A prefix is literal: a dot in it must not match any character.
{"sys.", []string{`{namespace="tenant-a"}`, `{namespace=~"sys\\..*",exported_namespace="tenant-a"}`}},
}
for _, tt := range tests {
if got := (MetricsConfig{PlatformNamespacePrefix: tt.prefix}).namespaceScope("tenant-a"); !slices.Equal(got, tt.want) {
t.Errorf("prefix %q: scope = %q, want %q", tt.prefix, got, tt.want)
}
}
}
func TestNamespaceScopeEscapesExportedNamespace(t *testing.T) {
for _, ns := range []string{`a"} or {namespace=~".+`, `back\slash`, "new\nline"} {
scope := MetricsConfig{PlatformNamespacePrefix: "cozy-"}.namespaceScope(ns)
filters, ok := parseSelector(scope[1])
if !ok || len(filters) != 2 || filters[1].label != "exported_namespace" || filters[1].value != ns {
t.Errorf("scope %q does not decode to exported_namespace=%q: %+v", scope[1], ns, filters)
}
}
}
// TestBuildMetricsParamsScopeOnEveryEndpoint checks that every endpoint of
// the namespace root sends the whole scope and nothing a client adds.
func TestBuildMetricsParamsScopeOnEveryEndpoint(t *testing.T) {
endpoints := map[string]url.Values{
"api/v1/query": {"query": {"up"}},
"api/v1/query_range": {"query": {"up"}, "start": {unix(testNow.Add(-5 * time.Minute))}, "end": {unix(testNow)}, "step": {"60"}},
"api/v1/series": {"match[]": {"up"}},
"api/v1/labels": {},
"api/v1/label/pod/values": {},
}
for _, prefix := range []string{"cozy-", ""} {
scope := MetricsConfig{PlatformNamespacePrefix: prefix}.namespaceScope("tenant-a")
for endpoint, in := range endpoints {
in = cloneValues(in)
in["extra_filters[]"] = []string{`{namespace=~".+"}`}
req, herr := BuildMetricsParams(endpoint, in, scope, testNow, testMetricsLimits())
if herr != nil {
t.Fatalf("%s: %v", endpoint, herr)
}
if got := req.params["extra_filters[]"]; !slices.Equal(got, scope) {
t.Errorf("prefix %q, %s: extra_filters[] = %q, want %q", prefix, endpoint, got, scope)
}
}
}
}
// An empty scope would send no extra_filters[] and read every namespace.
func TestBuildMetricsParamsRefusesEmptyScope(t *testing.T) {
if _, herr := BuildMetricsParams("api/v1/query", url.Values{"query": {"up"}}, nil, testNow, testMetricsLimits()); herr == nil || herr.Code != http.StatusInternalServerError {
t.Errorf("empty scope: %v, want 500", herr)
}
}
// TestBuildMetricsParamsCopiesScope guards against the forwarded parameters
// sharing the caller's slice.
func TestBuildMetricsParamsCopiesScope(t *testing.T) {
scope := testScope()
req, herr := BuildMetricsParams("api/v1/labels", url.Values{}, scope, testNow, testMetricsLimits())
if herr != nil {
t.Fatal(herr)
}
scope[1] = `{namespace=~".+"}`
if !slices.Equal(req.params["extra_filters[]"], wantMetricsScope) || !slices.Equal(req.params["match[]"], wantMetricsScope) {
t.Errorf("params follow the caller's slice: %v", req.params)
}
}
// TestVmselectSeesScopedSeries runs the parameters the proxy builds
// through a model of vmselect v1.136's series selection: repeated
// extra_filters[] are ORed (searchutil.GetExtraTagFilters), and every
// match[] or query selector is ANDed with each of them
// (searchutil.JoinTagFilterss, called from getCommonParamsInternal and
// promql eval). Regex filters match the whole value.
func TestVmselectSeesScopedSeries(t *testing.T) {
series := []struct {
labels map[string]string
visible bool
}{
{map[string]string{"namespace": "tenant-a"}, true},
{map[string]string{"namespace": "cozy-clickhouse-operator", "exported_namespace": "tenant-a", "chi": "clickhouse-db"}, true},
// A tenant pod claiming another namespace is not a platform exporter.
{map[string]string{"namespace": "tenant-y", "exported_namespace": "tenant-a"}, false},
{map[string]string{"namespace": "xcozy-a", "exported_namespace": "tenant-a"}, false},
{map[string]string{"namespace": "cozy-clickhouse-operator", "exported_namespace": "tenant-b"}, false},
{map[string]string{"namespace": "cozy-clickhouse-operator"}, false},
{map[string]string{"namespace": "tenant-b"}, false},
}
for _, endpoint := range []string{"api/v1/series", "api/v1/labels"} {
in := url.Values{}
if endpoint == "api/v1/series" {
in.Set("match[]", `{__name__=~".*"}`)
}
req := buildMetrics(t, endpoint, in)
for _, s := range series {
if got := vmselectSelects(t, req.params, s.labels); got != s.visible {
t.Errorf("%s: series %v visible = %v, want %v", endpoint, s.labels, got, s.visible)
}
}
}
// Without a prefix the exporter's series stay out of the namespace.
req, herr := BuildMetricsParams("api/v1/labels", url.Values{}, MetricsConfig{}.namespaceScope("tenant-a"), testNow, testMetricsLimits())
if herr != nil {
t.Fatal(herr)
}
if vmselectSelects(t, req.params, map[string]string{"namespace": "cozy-clickhouse-operator", "exported_namespace": "tenant-a"}) {
t.Error("exporter series visible with the platform prefix disabled")
}
}
func cloneValues(in url.Values) url.Values {
out := url.Values{}
for k, v := range in {
out[k] = slices.Clone(v)
}
return out
}
type labelFilter struct {
label, value string
regex bool
}
func (f labelFilter) matches(labels map[string]string) bool {
v := labels[f.label]
if f.regex {
return regexp.MustCompile("^(?:" + f.value + ")$").MatchString(v)
}
return v == f.value
}
// parseSelector reads the selectors the proxy and these tests write:
// {name="value",name=~"regex",...} with values quoted as MetricsQL reads
// them (strconv.Unquote in metricsql's extractStringValue).
func parseSelector(s string) ([]labelFilter, bool) {
body, ok := strings.CutPrefix(s, "{")
if !ok {
return nil, false
}
var out []labelFilter
for body != "}" {
i := strings.IndexAny(body, "=!")
if i <= 0 {
return nil, false
}
f := labelFilter{label: body[:i]}
body = body[i:]
switch {
case strings.HasPrefix(body, "=~"):
f.regex, body = true, body[2:]
case strings.HasPrefix(body, "="):
body = body[1:]
default:
return nil, false
}
lit, err := strconv.QuotedPrefix(body)
if err != nil {
return nil, false
}
if f.value, err = strconv.Unquote(lit); err != nil {
return nil, false
}
out = append(out, f)
body = strings.TrimPrefix(body[len(lit):], ",")
}
return out, true
}
func vmselectSelects(t *testing.T, params url.Values, labels map[string]string) bool {
t.Helper()
parse := func(sels []string) [][]labelFilter {
var out [][]labelFilter
for _, s := range sels {
f, ok := parseSelector(s)
if !ok {
t.Fatalf("cannot parse selector %q", s)
}
out = append(out, f)
}
return out
}
matches, extra := parse(params["match[]"]), parse(params["extra_filters[]"])
if len(matches) == 0 {
matches = [][]labelFilter{nil}
}
for _, m := range matches {
for _, e := range extra {
all := true
for _, f := range append(slices.Clone(m), e...) {
if f.label == "__name__" {
continue
}
all = all && f.matches(labels)
}
if all {
return true
}
}
}
return false
}
+1 -1
View File
@@ -297,7 +297,7 @@ func (s *Server) serveMetrics(w http.ResponseWriter, r *http.Request, route Rout
return herr
}
now := s.now()
req, herr := BuildMetricsParams(route.Endpoint, r.Form, route.Namespace, now, s.cfg.Metrics.Limits)
req, herr := BuildMetricsParams(route.Endpoint, r.Form, s.cfg.Metrics.namespaceScope(route.Namespace), now, s.cfg.Metrics.Limits)
if herr != nil {
return herr
}
@@ -177,7 +177,9 @@ func TestServerApplicationMetricsScope(t *testing.T) {
if got := r.form["match[]"]; len(got) != 1 || got[0] != lookups[r.path] {
t.Errorf("%s match[] = %q", r.path, got)
}
if got := r.form["extra_filters[]"]; !slices.Equal(got, []string{wantMetricsScope}) {
// lookupNames filters by the namespace selector alone, not by the
// namespace root's scope.
if got := r.form["extra_filters[]"]; !slices.Equal(got, []string{`{namespace="tenant-a"}`}) {
t.Errorf("%s extra_filters[] = %q", r.path, got)
}
// The query's own lookback is known and shorter than the 24h cap.
@@ -20,6 +20,7 @@ import (
"io"
"net/http"
"net/url"
"slices"
"sync"
"testing"
"time"
@@ -75,7 +76,7 @@ func TestServerMetricsRebuildsUpstreamRequest(t *testing.T) {
if r.path != "/api/v1/query_range" {
t.Errorf("upstream path = %s", r.path)
}
if got := r.form["extra_filters[]"]; len(got) != 1 || got[0] != wantMetricsScope {
if got := r.form["extra_filters[]"]; !slices.Equal(got, wantMetricsScope) {
t.Errorf("extra_filters[] = %q", got)
}
if r.form.Has("extra_filters") || r.form.Has("extra_label") || r.form.Has("nocache") {
@@ -105,7 +106,7 @@ func TestServerMetricsStoreByRange(t *testing.T) {
if want := []string{"shortterm", "longterm", "shortterm"}; len(h.stores) != 3 || h.stores[0] != want[0] || h.stores[1] != want[1] || h.stores[2] != want[2] {
t.Fatalf("stores = %v, want %v", h.stores, want)
}
if r := h.got[2]; r.path != "/api/v1/label/pod/values" || r.form.Get("match[]") != wantMetricsScope {
if r := h.got[2]; r.path != "/api/v1/label/pod/values" || !slices.Equal(r.form["match[]"], wantMetricsScope) {
t.Fatalf("label values upstream = %s %v", r.path, r.form)
}
}
@@ -93,6 +93,7 @@ func newPolicyHarness(t *testing.T, cfg Config, handler http.HandlerFunc, allow
cfg.Metrics.UpstreamURL = cfg.UpstreamURL
if cfg.Metrics.ShortTermStore == "" {
cfg.Metrics.ShortTermStore, cfg.Metrics.LongTermStore, cfg.Metrics.ShortTermRetention = "shortterm", "longterm", 72*time.Hour
cfg.Metrics.PlatformNamespacePrefix = "cozy-"
}
if cfg.Metrics.Limits == (MetricsLimits{}) {
cfg.Metrics.Limits = testMetricsLimits()
@@ -46,6 +46,7 @@ spec:
- --metrics-longterm-store={{ .Values.observabilityProxy.metrics.longtermStore }}
- --metrics-shortterm-retention={{ .Values.observabilityProxy.metrics.shorttermRetention }}
- --metrics-app-lookback={{ .Values.observabilityProxy.metrics.appLookback }}
- --metrics-platform-namespace-prefix={{ .Values.observabilityProxy.metrics.platformNamespacePrefix }}
ports:
- name: http
containerPort: 8080
@@ -63,6 +63,9 @@ tests:
- contains:
path: spec.template.spec.containers[0].args
content: --metrics-app-lookback=720h
- contains:
path: spec.template.spec.containers[0].args
content: --metrics-platform-namespace-prefix=cozy-
- it: follows renamed metrics stores
set:
@@ -85,6 +88,19 @@ tests:
path: spec.template.spec.containers[0].args
content: --metrics-app-lookback=48h
# An empty value must reach the binary as an explicit empty flag; a
# dropped flag would fall back to the cozy- default.
- it: passes an empty platform namespace prefix through
set:
observabilityProxy.metrics.platformNamespacePrefix: ""
asserts:
- contains:
path: spec.template.spec.containers[0].args
content: --metrics-platform-namespace-prefix=
- notContains:
path: spec.template.spec.containers[0].args
content: --metrics-platform-namespace-prefix=cozy-
- it: falls back to cozy.local when the platform passes no cluster domain
asserts:
- contains:
@@ -17,6 +17,10 @@ observabilityProxy:
# How far before its window an application metrics query may read;
# one reading further back answers 422 on the application root.
appLookback: 720h
# Exporters in namespaces with this prefix may attribute a series to a
# tenant namespace through exported_namespace, and the series is then
# part of that namespace's metrics. Empty disables it.
platformNamespacePrefix: cozy-
resources:
requests:
cpu: 10m