Resolve the overlap with the per-application roots: the namespace root
keeps its platform exporter branch, the application root still replaces
the whole scope with its own selectors, and the label APIs mark the
repeated scope so that replacement reaches match[] as well.
Assisted-by: LLM
Signed-off-by: Maxim Belyy <maximbel2003@gmail.com>
## What this PR does
Depends on #5219. This PR is stacked on that branch; the description
covers only what it adds on top.
Adds the per-application roots to the observability proxy, as described
in the design proposals [Tenant application
logs](https://github.com/cozystack/community/pull/95) and [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94):
```text
/apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/log/select/logsql/<endpoint>
/apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/metrics/api/v1/<endpoint>
```
This lets the console show the logs and resource consumption of one
application rather than of the whole namespace.
What is added:
- Plural-to-Kind mapping from an informer over `ApplicationDefinitions`.
The Kind is looked up only after the `SubjectAccessReview` (`get
<plural>/log` or `get <plural>/metrics`), so an unauthenticated caller
cannot probe which plurals exist. An unknown plural answers `404`; an
unsynced cache or a plural declared by two kinds answers `503`.
- Logs: a second `extra_stream_filters` value with the application Kind
and name, on top of the unchanged namespace filter. VictoriaLogs ANDs
the two, on every endpoint including `tail`.
- Metrics: the forced `extra_filters[]` becomes an OR of three branches,
each still bound to the namespace:
1. series carrying the application identity labels;
2. series of the application's pods, found through `kube_pod_labels`;
3. series of the application's PVCs, found through
`kube_persistentvolumeclaim_labels`.
- The pod and PVC lookups cap the number of names
(`--metrics-app-max-pods`, `--metrics-app-max-pvcs`); over the cap
answers `422`. A failed pod or PVC lookup fails the request (`502`, or
`504` on the deadline) rather than answer without the application's
series, and so does a lookup that vmselect marks `isPartial`, since a
storage node that did not answer leaves names out (a partial list
already over the cap still answers `422`); vmselect's series-limit
refusal of a lookup is passed through as `422`. A query that reads
further back than `--metrics-app-lookback` (default 720h, chart value
`observabilityProxy.metrics.appLookback`), or by an unknown amount, and
a query whose samples lie at a time the lookup cannot cover answer `422`
on the application root; this includes every query with an `@` modifier,
`WITH` with a negative offset, and offsets that are not plain durations.
The namespace root still serves them. `rollup_candlestick`, a backslash
or a `#` comment outside a string, an offset directly followed by a
minus (compound durations such as `-1h-30m` or `1h-2h`), and a subquery
step that is not a plain duration are refused as well. The lookup
reaches past the evaluation time by any negative offset and by two steps
per subquery level.
- Application routes require read access to the namespace root as well
as the application subresource; a grant narrowed with `resourceNames`
alone answers `403`.
- The proxy's ClusterRole gains `list`/`watch` on
`cozystack.io/applicationdefinitions`. No tenant RBAC change is needed:
the `view` role already allows `get` on every `apps.cozystack.io`
resource and subresource.
- `/readyz` now also waits for the `ApplicationDefinitions` cache, so a
missing CRD or RBAC rule keeps the whole proxy unready, namespace roots
included.
#### Data comes from the identity labels
The application roots find data only through identity labels added by
three PRs that are now in main:
- #5215 `feat(monitoring-agents)`: export PVC labels from
kube-state-metrics (PVC branch).
- #5216 `feat(monitoring)`: stamp application identity on scraped series
(identity branch).
- #5217 `feat(monitoring-agents)`: keep application identity in
container log streams (logs root).
Merge after #5219.
The pod branch does not depend on them, because `kube_pod_labels`
already exports all pod labels. Logs and series written before those
changes rolled out carry no identity and are reachable only through the
namespace root.
#### Security condition: a convenience filter, not an isolation boundary
The application roots narrow the namespace roots; they do not isolate
applications from each other. Each application route therefore requires
two grants: the application subresource (`get <plural>/log` or `get
<plural>/metrics`) and read access to the namespace root (`list
cozystack.io/logs` or `cozystack.io/metrics` in the namespace). A grant
narrowed with `resourceNames` alone is refused. Every tenant role that
can read applications already has both. The application scope cannot
isolate yet because:
1. The lineage webhook keeps `apps.cozystack.io/application.*` labels
that a user sets on a pod or PVC that has no application owner, so such
an object can claim another application's identity.
2. The webhook's `objectSelector` skips any object created with
`internal.cozystack.io/managed-by-cozystack`, pods included, so forged
labels on such an object are never examined.
3. The pod and PVC branches match by name over the lookup window, so an
object that reuses the name of another application's recently deleted
pod or PVC pulls that object's older series into scope.
The proposed fix (a separate change to the lineage webhook) is to strip
the identity labels when no owner resolves, and to stop skipping CREATE
requests for objects carrying the managed-by label.
#### Other known gaps
- PVCs created from a StatefulSet's `volumeClaimTemplates` usually get
no lineage labels, so their volume series are not in the application
scope.
- ingress-nginx series intentionally carry no application identity, so
they are reachable only through the namespace root.
- A request for an application that does not exist answers `200` with an
empty result rather than `404`: the proxy checks authorization, not
existence. Nothing leaks; whether a `404` is wanted is open for review.
#### How it was tested
- `go test -race` and `go vet` for the proxy packages, including a
narrowed-grant test (access to application `a` only): every application
path for `b` answers `403` and no request reaches the store. Name
escaping in the pod and PVC branches is covered for regex metacharacters
and quotes.
- helm-unittest for `observability-proxy` (new RBAC rule).
- Verified on a test cluster, with the three identity PRs above applied:
the application logs root of a Redis instance returned only that
instance's lines and none from a neighbouring pod in the same namespace;
its application metrics root returned only the Redis pods' series.
### Screenshots
No UI changes.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
Downstream: no trigger matched. Candidate: none affected, because the
diff on top of #5219 is confined to the proxy's Go code and its own
chart RBAC; it reads `ApplicationDefinitions` but does not change their
semantics or the CRD. The security condition above may deserve a note in
the design proposals in `cozystack/community`, which is not a
trigger-map item.
### Release note
```release-note
feat(observability-proxy): add per-application logs and metrics roots under `/apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/{log,metrics}/`, scoped by application identity labels. Data appears for logs and series collected after the application identity changes to Fluent Bit, VMAgent and kube-state-metrics are rolled out.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **New Features**
* Added a tenant-aware observability proxy for logs and metrics, with
namespace and application scoping, access checks, and per-user request
limits.
* Enabled the proxy in the system package bundle and routed console log
and metrics requests through it.
* Added live log streaming and health, readiness, and metrics endpoints.
* **Bug Fixes**
* Invalid or excessive query parameters are rejected; requests with
uncertain application-metric scope are refused rather than returning
incomplete results.
* **Tests**
* Added end-to-end and integration coverage for tenant isolation,
console routing, access controls, and log streaming.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
This clears the golangci-lint findings on main and fixes the errors
several of them pointed at. With the default linter set (errcheck,
govet, ineffassign, staticcheck, unused), main had 197 findings when
this branch started. Rebased on main at 0f169540, the branch leaves 38,
all deprecations that need a design decision rather than a lint fix;
they are listed below.
Most commits are mechanical. They remove unused helpers and constants,
reword error strings, take the staticcheck simplifications, discard
best-effort cleanup errors explicitly, and move to the FieldsV1 and ptr
accessors. A loop in pkg/lineage that always ended in its first pass
becomes a helper, with a regression test for each fallback. These change
behaviour:
- Lost output is now a failure for the commands this PR touches: cozypkg
list, the validation report of validate, push and tap, the push status
line, the tap summary, and check-readiness. The command exits non-zero
and says whether the action it reports on already happened. A closed
pipe ends the process with SIGPIPE, as for other Unix tools.
- backport-batch writes its output before it pushes or publishes. Once a
write to stdout or stderr fails, the run changes nothing more: no push,
no GitHub write, no local branch move, and no new pick. A pick already
in flight still finishes its rollback. The run exits 2, and it catches
SIGPIPE, so a closed pipe is handled the same way. A confirmation
printed after an action is lost only when stderr itself fails, so a lost
confirmation can only be reported by the exit code.
- The SSE server drops a client as soon as a write to it fails. A stream
still open at shutdown no longer panics on a double close, the handler
waits for its pump before returning, and a stream that opens during
shutdown gets 503.
- The kube-ovn plunger now fails its manager when the status server
cannot bind :18080, as a metrics server bind failure does. Before, it
ran without the endpoint and logged nothing.
- A timed-out check-readiness --wait still ends with its verdict, and
the help lists the exit statuses.
The 37 findings left are SA1019 deprecations, kept out of this PR on
purpose:
- GetEventRecorderFor (12, in cmd/*/main.go). The events.k8s.io recorder
has a different interface and needs RBAC for that API group. It changes
what events look like, so it needs its own design.
- client.Apply (12, one more site arrived from main). Moving to
Client.Apply with apply configurations changes how every server-side
apply call is built and how the fake client in tests handles it. It
overlaps the dependency rollup in #4869 and should come after it.
- Result.Requeue (10). Requeue retries with rate-limited backoff,
RequeueAfter with a fixed delay. The interval has to come from whoever
owns each controller.
- scheme.Builder (3, in api/*). Replacing it changes the type of the
exported SchemeBuilder that other repositories import.
- cel.Functions (1, in pkg/reservation). It stays for its late binding,
as the comment next to it explains.
Each behaviour change has tests that failed before the fix. I ran go
test on every changed package (for cmd/backport-batch, the tests around
output and publishing, since the full package takes about ten minutes),
go build ./..., go vet on the changed packages, and a final
golangci-lint run with uncapped output on the rebased tree:
`golangci-lint run --verbose --max-same-issues=0
--max-issues-per-linter=0`, which reports the 38 findings above and
nothing else.
The tests in pkg/lineage, pkg/ovnstatus, pkg/reservation, cmd/cozypkg
and cmd/check-readiness now run in CI: a separate commit adds those
packages to go-unit-tests. Some packages with tests are still in no CI
target: cmd/cozystack-controller, cmd/cozystack-operator,
cmd/observability-proxy, pkg/apiserver, pkg/apis/apps/defaulting,
pkg/apis/apps/install, pkg/apis/apps/presets, pkg/apis/apps/validation,
pkg/apis/core/install, pkg/apis/sdn/install and pkg/version. That
predates this PR.
One known nit: when backport-batch refuses an action after its output
was lost, stderr can show two "output was lost" lines.
### Screenshots
No UI change.
### Downstream repositories
I walked the trigger map against the diff. cozypkg counts as developer
tooling, but only its error paths changed, not a flag, its usage or its
output format, so the development page needs no update. Telemetry metric
names and labels are unchanged.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
chore: cozypkg, check-readiness and backport-batch now fail when their output cannot be written, and backport-batch pushes and publishes nothing once that happens. The kube-ovn plunger fails when its status server cannot start, drops status clients whose writes fail, and no longer panics when it shuts down with a client connected.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Bug Fixes**
* Commands now report output failures instead of silently succeeding.
Backport operations stop before further external changes when required
output cannot be written; readiness checks return a failure status when
output writing fails.
* Package listings, validation reports, push status messages, and tap
summaries now report output errors. Tap commands indicate when changes
were applied before a summary could be written.
* Live event streams handle failed writes and shutdown more reliably,
and status-server startup errors are reported.
* **Documentation**
* Readiness-check help now explains exit statuses and closed-pipe
behavior.
* **Improvements**
* Error messages for several backup and inventory operations now provide
clearer context.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The drain tests now assert that RunThenDrain returns the run's nil
error, the idle-tail test fails on any error other than the stream
ending or being cut, and a test fixture file fails the test when it
cannot be closed.
Removing a temporary directory and closing a response body that has
been read are best-effort: a failure leaves nothing for the caller to
act on and does not change any result, so those errors are discarded
explicitly. The same holds for the stub servers' response writes, whose
failure surfaces on the client side of the test.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A platform exporter such as the ClickHouse operator's runs in its own
cozy-* namespace and reports on objects in tenant namespaces. vmagent
labels its series with the exporter's namespace and moves the
namespace the exporter reported to exported_namespace, so the
namespace root, scoped to namespace="<ns>" alone, never returned them.
The scope gains a second extra_filters[] value matching
exported_namespace="<ns>" only on series scraped from a namespace with
the platform prefix. A bare exported_namespace condition would let any
tenant pod that writes a namespace label inject series into another
tenant's namespace root. vmselect ORs the two values, and the label
APIs repeat both in match[], where the match[] and extra_filters[]
cross join still selects exactly their union. An empty prefix keeps
the previous single-value scope.
Assisted-by: LLM
Signed-off-by: Maxim Belyy <maximbel2003@gmail.com>
The application roots asked only for the application subresource. Their
pod and PVC branches select by namespace and name across the whole
lookup window, so a caller granted only application "a" received the
older series of application "b" whenever "b" had used one of the same
pod or PVC names. The lineage labels the scope rests on can also be set
by whoever creates an unowned pod.
Until the scope carries application identity on every branch, the
application roots also require read access to the namespace root, the
same review the namespace root asks. Every tenant role that reads
applications already holds it, so no current grant loses access; a
grant narrowed by resourceNames alone is refused instead of trusted.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Graphs of one application need its series without the caller knowing
which pods and volumes it ran on. Exporter series carry the identity
the scrape class stamps, but cAdvisor and kubelet volume series do not,
so the scope ORs three branches: the identity, and the application's
pods and PVCs as kube-state-metrics labelled them over the window. Each
branch also carries the namespace, so the scope is never wider than the
namespace root.
The lookup reaches back by the query's own lookback, bounded by a flag,
so pods replaced inside an expression's range still count. A failed
lookup only drops its branch. A list over its cap answers 422 instead
of being truncated, which would hide series without saying so.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
A viewer of one application gets that application's lines without
writing a namespace query and filtering by hand, and a grant narrowed to
one application by resourceNames is honoured on every logs endpoint,
tail and the pickers included. The namespace filter stays and the
application's identity is ANDed next to it as a second stream filter,
so the result can only be narrower than the namespace root.
The plural is mapped to its kind only after the caller is authorized,
so an anonymous client cannot probe which applications exist, and the
proxy is not ready until the ApplicationDefinitions cache is.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The namespace metrics root shares the logs root's authorization path:
the SubjectAccessReview comes from the path alone, so a POST query,
which Grafana sends by default, is authorized exactly like a GET. Both
roots also share the per-user rate and concurrency limits and the way
an unreachable store (502) is told apart from a slow one (504); the
upstream call moves into one helper so the two stay identical.
Every namespace resolves to the root tenant's vmselect for now,
because the series the graphs need are only scraped there.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The test only checked that a Retry-After was present, while the rate
limit test pins its value, so a change to the promised hint passed.
It now expects the literal value. The note on the empty fields_limit
case now gives the reason it is refused.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Parsing the scheme with a case-insensitive split changed which header
forms reach the token review; a bare scheme, an empty token and a
scheme run into the token must still answer 401.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Only the rate-limit 429 said when to retry. A concurrency or tail 429
came without it, so a client had no hint and could retry at once in a
loop. It now carries a one-second hint; when a slot frees depends on
the user's other requests, which the proxy cannot predict.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The authorization scheme is case-insensitive, and kube-apiserver
accepts "bearer" as well as "Bearer", so a client that works against
the API server got 401 from the proxy with the same header.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Only tails opt out of the transport's gzip negotiation; bounded
requests keep it on purpose. Assert that side too, so a change that
applies identity to every request fails a test instead of silently
dropping compression between the proxy and the store.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Go's transport asks for gzip on its own when a request sets no
Accept-Encoding. vlselect answers a gzip-negotiated tail through a gzip
writer that holds even the response headers until the first line, so the
proxy could not forward its 200 and the console kept showing a
connecting state for quiet applications; buffered lines were also
delayed.
The tail request now sends Accept-Encoding: identity, which also turns
off the transport's transparent gzip. Other endpoints keep it: they are
bounded responses read to the end, compression saves bandwidth between
the proxy and the store, and the transport decompresses before the body
is copied to the client.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Clients such as Grafana and browser retry logic back off on 429 Too
Many Requests, not on 422, which they treat as a final rejection of
the query. The per-user rate limit, the per-user concurrency limit and
the open-tail caps are about the caller's pace, not the query, so they
now answer 429. The rate limiter knows when its bucket refills, so that
answer carries Retry-After; the concurrency caps cannot know when a
slot frees and omit it. Range, step, bucket and limit violations keep
422.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
A request that hits the proxy's own deadline was answered like an
unreachable store, so a heavy query looked like an outage; it now
answers 504, and a client that went away is no longer logged as a
store failure. A tail line is read through a fixed buffer, so a single
huge log line cannot make one stream hold unbounded memory.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The console nginx selects its location on the normalized path but
forwards the original request URI. A path such as
namespaces/tenant-b/../tenant-a/logs/ reaches the proxy as written, so
any interpretation the proxy makes of dot segments, empty segments or
percent-encoding could make the namespace in the SubjectAccessReview
differ from the one a normalizer downstream would see. Such paths are
answered 400 instead of being interpreted.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Tenants get a way to read their namespace's logs without Grafana and
without reaching vlselect directly, which would give them neither an
RBAC check nor a forced scope. The handler authenticates, authorizes
from the path, rebuilds the upstream request with fixed tenant headers
and no client headers, and returns VictoriaLogs' native response.
A live tail is flushed after every line, is not subject to the request
timeout, and is closed once idle, so it survives far past the apiserver
request timeout without leaving forgotten browser tabs holding streams
open. An unreachable store answers 502 so the dashboard can tell an
outage from an empty result.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>