Files
cozystack/internal
Aleksei Sviridkin 17bbca2088 feat(observability-proxy): serve tenant namespace metrics (#5219)
## What this PR does

Depends on #5214. This PR is stacked on that branch; the description
covers only what it adds on top.

Backward compatibility: this adds a new package and a new HelmRelease;
nothing existing is renamed, removed or changed, so no migration is
needed.

Adds the namespace metrics root to the observability proxy, as described
in the design proposal [Tenant resource-consumption
metrics](https://github.com/cozystack/community/pull/94). A tenant user
can query the metrics of their own namespace from VictoriaMetrics with
their Kubernetes credentials, without seeing other tenants' series.

What is added:

- Serves
`/apis/cozystack.io/v1alpha1/namespaces/<ns>/metrics/api/v1/{query,query_range,series,labels,label/<name>/values}`,
GET and POST. Anything else under the root answers `404`.
- Authorization by one `SubjectAccessReview` (`list` on
`cozystack.io/metrics` in the namespace); the aggregated tenant `view`
role gains that rule.
- The upstream request is rebuilt from an allowlist (`query`, `start`,
`end`, `step`, `time`, `match[]`, `timeout`). The scope is a single
forced `extra_filters[]={namespace="<ns>"}`; client-supplied
`extra_filters`, `extra_filters[]` and `extra_label` are dropped,
because vmselect would OR them with the forced filter. The query itself
is not rewritten.
- Label endpoints always carry a `match[]` (the namespace scope when the
client sent none), so they stay scoped even if vmselect's
`-search.ignoreExtraFiltersAtLabelsAPI` is ever enabled.
- Cost limits: maximum range (30 days), maximum points (11000), and a
clamped `timeout` (30 seconds); query-shape limits answer `422`, the
concurrency limit answers `429`. Times are accepted as unix seconds or
RFC3339; durations are rounded up to whole milliseconds before limits
are checked.
- Store selection between the short-term and long-term VictoriaMetrics
stores, configured by new chart values (`observabilityProxy.metrics`).

#### Store selection: a known limitation and an open question

The proxy has to send each query to the short-term store (3 days of
retention, finer deduplication) or the long-term store. The choice
depends on how far back the query reads, which is not only the request
window but also the windows, subquery steps and offsets inside the
expression.

The current implementation estimates that lookback with a conservative
scan of the expression text, not with a parser. It sums every `[...]`
window and subquery step, every positive `offset`, and allowances for
implicit subqueries and vmselect's default step and latency offset.
Anything it cannot read with certainty (`@`, `WITH` templates, step
multiples such as `5i`, template variables, unusual duration syntax)
goes to the long-term store. The query is forwarded unchanged, so the
estimate affects only the store choice, never the scope.

This is a heuristic. It is designed to err only towards the long-term
store, which holds the same recent data at coarser deduplication, so a
wrong guess returns coarser data rather than missing data in most cases.
A very short `rate` window sent to the long-term store can still come
back empty. Two review rounds each found new expressions the scan
underestimated, which were then fixed; there may be more.

Open question for reviewers: should the scan be replaced by parsing the
expression with `github.com/VictoriaMetrics/metricsql` (the library
vmselect itself uses), only for the store choice? That gives an exact
lookback at the cost of a new Go dependency. The namespace scope would
still be enforced by vmselect through the single `extra_filters[]`, so
the isolation design does not change either way.

A user-visible consequence of counting the lookback: a range whose start
sits right at the short-term retention edge (for example "last 3 days")
now goes to the long-term store.

#### How it was tested

- `go test -race` and `go vet` for the proxy packages; table tests for
the lookback estimate, including mutation checks of the `@` and
unparseable-window cases.
- helm-unittest for `observability-proxy` (new args and the tenant role
rule).
- Verified on a test cluster: `401` and `403` where expected; results
contain only the caller's namespace; a query naming another tenant's
namespace and a client-supplied `extra_filters[]` return nothing extra;
`query_range`, `label/<name>/values` and `series` are scoped; an admin
endpoint answers `404`.
- Not verified: the limits under load, and the store choice on windows
longer than 3 days (the test cluster had less than an hour of data).

#### Known limitations

- Relative times (`start=-1h`, `end=now`) are refused with `400`, while
the logs root accepts durations. The console sends unix seconds, so it
is not affected.
- The proxy bounds the evaluation window, the step, the number of points
of `query_range` and the timeout, not how far an expression reads:
windows and offsets inside the expression (`up[10y]`, `x[30d:1ms]`,
`offset 20d`) are bounded only by the store's retention and vmselect's
own limits.
- Times and durations are sent in a form vmselect parses exactly
(RFC3339 with milliseconds for times, durations that truncate to the
checked milliseconds); times outside 1970..2262 are refused with `400`.
A test repeats vmselect's parsing of the exact strings the proxy sends.
- `--metrics-max-range` defaults to 30 days while the longterm store
keeps 14, so the older part of a long range comes back empty rather than
refused. The two should be aligned or documented.
- The store estimate errs towards longterm: deeply nested expressions
with a large step can go to longterm although shortterm holds the data,
and longterm deduplicates at 5 minutes, so short windows such as
`rate(x[1m])` can come back sparse there.
- The `namespace` label is only as trustworthy as the writer. Any pod in
a child tenant may write to the vminsert of every ancestor tenant
(`packages/apps/tenant/templates/networkpolicy.yaml`), and vminsert does
not authenticate writers, so a child tenant can write series labelled
with another tenant's namespace. Through this proxy the owner of that
namespace would see those series as its own, and a flood of them could
push its queries over the series limit. Reading another tenant's data is
not possible this way. The write path predates this PR and already
affects the root tenant's dashboards and alerts; closing it needs the
write side to mark untrusted series, for example a vmauth in front of
vminsert that forces an `extra_label` the proxy then excludes.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: no trigger matched. Candidate: none affected, because the
diff on top of #5214 is confined to the proxy's Go code and its own
chart under `packages/system/observability-proxy` (no platform values,
no package under `packages/apps/`, no CRD, no `hack/` change). A
user-facing page for the metrics API on the website would be useful but
is not required by the trigger map.

### Release note

```release-note
feat(observability-proxy): let tenant users query their namespace metrics from VictoriaMetrics with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/metrics/api/v1/`. Access requires the tenant `view` role or higher. Ranges are limited to 30 days and 11000 points.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* The observability proxy now supports tenant-scoped metrics queries
alongside logs.
* Metrics requests are routed to short- or long-term stores based on
their time range, with configurable retention and query limits.
* Metrics access is subject to tenant authorization, request validation,
and concurrency limits.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 16:26:22 +03:00
..