mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 20:23:39 +09:00
## What this PR does Depends on #5214. This PR is stacked on that branch; the description covers only what it adds on top. Backward compatibility: this adds a new package and a new HelmRelease; nothing existing is renamed, removed or changed, so no migration is needed. Adds the namespace metrics root to the observability proxy, as described in the design proposal [Tenant resource-consumption metrics](https://github.com/cozystack/community/pull/94). A tenant user can query the metrics of their own namespace from VictoriaMetrics with their Kubernetes credentials, without seeing other tenants' series. What is added: - Serves `/apis/cozystack.io/v1alpha1/namespaces/<ns>/metrics/api/v1/{query,query_range,series,labels,label/<name>/values}`, GET and POST. Anything else under the root answers `404`. - Authorization by one `SubjectAccessReview` (`list` on `cozystack.io/metrics` in the namespace); the aggregated tenant `view` role gains that rule. - The upstream request is rebuilt from an allowlist (`query`, `start`, `end`, `step`, `time`, `match[]`, `timeout`). The scope is a single forced `extra_filters[]={namespace="<ns>"}`; client-supplied `extra_filters`, `extra_filters[]` and `extra_label` are dropped, because vmselect would OR them with the forced filter. The query itself is not rewritten. - Label endpoints always carry a `match[]` (the namespace scope when the client sent none), so they stay scoped even if vmselect's `-search.ignoreExtraFiltersAtLabelsAPI` is ever enabled. - Cost limits: maximum range (30 days), maximum points (11000), and a clamped `timeout` (30 seconds); query-shape limits answer `422`, the concurrency limit answers `429`. Times are accepted as unix seconds or RFC3339; durations are rounded up to whole milliseconds before limits are checked. - Store selection between the short-term and long-term VictoriaMetrics stores, configured by new chart values (`observabilityProxy.metrics`). #### Store selection: a known limitation and an open question The proxy has to send each query to the short-term store (3 days of retention, finer deduplication) or the long-term store. The choice depends on how far back the query reads, which is not only the request window but also the windows, subquery steps and offsets inside the expression. The current implementation estimates that lookback with a conservative scan of the expression text, not with a parser. It sums every `[...]` window and subquery step, every positive `offset`, and allowances for implicit subqueries and vmselect's default step and latency offset. Anything it cannot read with certainty (`@`, `WITH` templates, step multiples such as `5i`, template variables, unusual duration syntax) goes to the long-term store. The query is forwarded unchanged, so the estimate affects only the store choice, never the scope. This is a heuristic. It is designed to err only towards the long-term store, which holds the same recent data at coarser deduplication, so a wrong guess returns coarser data rather than missing data in most cases. A very short `rate` window sent to the long-term store can still come back empty. Two review rounds each found new expressions the scan underestimated, which were then fixed; there may be more. Open question for reviewers: should the scan be replaced by parsing the expression with `github.com/VictoriaMetrics/metricsql` (the library vmselect itself uses), only for the store choice? That gives an exact lookback at the cost of a new Go dependency. The namespace scope would still be enforced by vmselect through the single `extra_filters[]`, so the isolation design does not change either way. A user-visible consequence of counting the lookback: a range whose start sits right at the short-term retention edge (for example "last 3 days") now goes to the long-term store. #### How it was tested - `go test -race` and `go vet` for the proxy packages; table tests for the lookback estimate, including mutation checks of the `@` and unparseable-window cases. - helm-unittest for `observability-proxy` (new args and the tenant role rule). - Verified on a test cluster: `401` and `403` where expected; results contain only the caller's namespace; a query naming another tenant's namespace and a client-supplied `extra_filters[]` return nothing extra; `query_range`, `label/<name>/values` and `series` are scoped; an admin endpoint answers `404`. - Not verified: the limits under load, and the store choice on windows longer than 3 days (the test cluster had less than an hour of data). #### Known limitations - Relative times (`start=-1h`, `end=now`) are refused with `400`, while the logs root accepts durations. The console sends unix seconds, so it is not affected. - The proxy bounds the evaluation window, the step, the number of points of `query_range` and the timeout, not how far an expression reads: windows and offsets inside the expression (`up[10y]`, `x[30d:1ms]`, `offset 20d`) are bounded only by the store's retention and vmselect's own limits. - Times and durations are sent in a form vmselect parses exactly (RFC3339 with milliseconds for times, durations that truncate to the checked milliseconds); times outside 1970..2262 are refused with `400`. A test repeats vmselect's parsing of the exact strings the proxy sends. - `--metrics-max-range` defaults to 30 days while the longterm store keeps 14, so the older part of a long range comes back empty rather than refused. The two should be aligned or documented. - The store estimate errs towards longterm: deeply nested expressions with a large step can go to longterm although shortterm holds the data, and longterm deduplicates at 5 minutes, so short windows such as `rate(x[1m])` can come back sparse there. - The `namespace` label is only as trustworthy as the writer. Any pod in a child tenant may write to the vminsert of every ancestor tenant (`packages/apps/tenant/templates/networkpolicy.yaml`), and vminsert does not authenticate writers, so a child tenant can write series labelled with another tenant's namespace. Through this proxy the owner of that namespace would see those series as its own, and a flood of them could push its queries over the series limit. Reading another tenant's data is not possible this way. The write path predates this PR and already affects the root tenant's dashboards and alerts; closing it needs the write side to mark untrusted series, for example a vmauth in front of vminsert that forces an `extra_label` the proxy then excludes. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Downstream: no trigger matched. Candidate: none affected, because the diff on top of #5214 is confined to the proxy's Go code and its own chart under `packages/system/observability-proxy` (no platform values, no package under `packages/apps/`, no CRD, no `hack/` change). A user-facing page for the metrics API on the website would be useful but is not required by the trigger map. ### Release note ```release-note feat(observability-proxy): let tenant users query their namespace metrics from VictoriaMetrics with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/metrics/api/v1/`. Access requires the tenant `view` role or higher. Ranges are limited to 30 days and 11000 points. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * The observability proxy now supports tenant-scoped metrics queries alongside logs. * Metrics requests are routed to short- or long-term stores based on their time range, with configurable retention and query limits. * Metrics access is subject to tenant authorization, request validation, and concurrency limits. <!-- end of auto-generated comment: release notes by coderabbit.ai -->