## What this PR does
Depends on #5219. This PR is stacked on that branch; the description
covers only what it adds on top.
Adds the per-application roots to the observability proxy, as described
in the design proposals [Tenant application
logs](https://github.com/cozystack/community/pull/95) and [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94):
```text
/apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/log/select/logsql/<endpoint>
/apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/metrics/api/v1/<endpoint>
```
This lets the console show the logs and resource consumption of one
application rather than of the whole namespace.
What is added:
- Plural-to-Kind mapping from an informer over `ApplicationDefinitions`.
The Kind is looked up only after the `SubjectAccessReview` (`get
<plural>/log` or `get <plural>/metrics`), so an unauthenticated caller
cannot probe which plurals exist. An unknown plural answers `404`; an
unsynced cache or a plural declared by two kinds answers `503`.
- Logs: a second `extra_stream_filters` value with the application Kind
and name, on top of the unchanged namespace filter. VictoriaLogs ANDs
the two, on every endpoint including `tail`.
- Metrics: the forced `extra_filters[]` becomes an OR of three branches,
each still bound to the namespace:
1. series carrying the application identity labels;
2. series of the application's pods, found through `kube_pod_labels`;
3. series of the application's PVCs, found through
`kube_persistentvolumeclaim_labels`.
- The pod and PVC lookups cap the number of names
(`--metrics-app-max-pods`, `--metrics-app-max-pvcs`); over the cap
answers `422`. A failed pod or PVC lookup fails the request (`502`, or
`504` on the deadline) rather than answer without the application's
series, and so does a lookup that vmselect marks `isPartial`, since a
storage node that did not answer leaves names out (a partial list
already over the cap still answers `422`); vmselect's series-limit
refusal of a lookup is passed through as `422`. A query that reads
further back than `--metrics-app-lookback` (default 720h, chart value
`observabilityProxy.metrics.appLookback`), or by an unknown amount, and
a query whose samples lie at a time the lookup cannot cover answer `422`
on the application root; this includes every query with an `@` modifier,
`WITH` with a negative offset, and offsets that are not plain durations.
The namespace root still serves them. `rollup_candlestick`, a backslash
or a `#` comment outside a string, an offset directly followed by a
minus (compound durations such as `-1h-30m` or `1h-2h`), and a subquery
step that is not a plain duration are refused as well. The lookup
reaches past the evaluation time by any negative offset and by two steps
per subquery level.
- Application routes require read access to the namespace root as well
as the application subresource; a grant narrowed with `resourceNames`
alone answers `403`.
- The proxy's ClusterRole gains `list`/`watch` on
`cozystack.io/applicationdefinitions`. No tenant RBAC change is needed:
the `view` role already allows `get` on every `apps.cozystack.io`
resource and subresource.
- `/readyz` now also waits for the `ApplicationDefinitions` cache, so a
missing CRD or RBAC rule keeps the whole proxy unready, namespace roots
included.
#### Data comes from the identity labels
The application roots find data only through identity labels added by
three PRs that are now in main:
- #5215 `feat(monitoring-agents)`: export PVC labels from
kube-state-metrics (PVC branch).
- #5216 `feat(monitoring)`: stamp application identity on scraped series
(identity branch).
- #5217 `feat(monitoring-agents)`: keep application identity in
container log streams (logs root).
Merge after #5219.
The pod branch does not depend on them, because `kube_pod_labels`
already exports all pod labels. Logs and series written before those
changes rolled out carry no identity and are reachable only through the
namespace root.
#### Security condition: a convenience filter, not an isolation boundary
The application roots narrow the namespace roots; they do not isolate
applications from each other. Each application route therefore requires
two grants: the application subresource (`get <plural>/log` or `get
<plural>/metrics`) and read access to the namespace root (`list
cozystack.io/logs` or `cozystack.io/metrics` in the namespace). A grant
narrowed with `resourceNames` alone is refused. Every tenant role that
can read applications already has both. The application scope cannot
isolate yet because:
1. The lineage webhook keeps `apps.cozystack.io/application.*` labels
that a user sets on a pod or PVC that has no application owner, so such
an object can claim another application's identity.
2. The webhook's `objectSelector` skips any object created with
`internal.cozystack.io/managed-by-cozystack`, pods included, so forged
labels on such an object are never examined.
3. The pod and PVC branches match by name over the lookup window, so an
object that reuses the name of another application's recently deleted
pod or PVC pulls that object's older series into scope.
The proposed fix (a separate change to the lineage webhook) is to strip
the identity labels when no owner resolves, and to stop skipping CREATE
requests for objects carrying the managed-by label.
#### Other known gaps
- PVCs created from a StatefulSet's `volumeClaimTemplates` usually get
no lineage labels, so their volume series are not in the application
scope.
- ingress-nginx series intentionally carry no application identity, so
they are reachable only through the namespace root.
- A request for an application that does not exist answers `200` with an
empty result rather than `404`: the proxy checks authorization, not
existence. Nothing leaks; whether a `404` is wanted is open for review.
#### How it was tested
- `go test -race` and `go vet` for the proxy packages, including a
narrowed-grant test (access to application `a` only): every application
path for `b` answers `403` and no request reaches the store. Name
escaping in the pod and PVC branches is covered for regex metacharacters
and quotes.
- helm-unittest for `observability-proxy` (new RBAC rule).
- Verified on a test cluster, with the three identity PRs above applied:
the application logs root of a Redis instance returned only that
instance's lines and none from a neighbouring pod in the same namespace;
its application metrics root returned only the Redis pods' series.
### Screenshots
No UI changes.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
Downstream: no trigger matched. Candidate: none affected, because the
diff on top of #5219 is confined to the proxy's Go code and its own
chart RBAC; it reads `ApplicationDefinitions` but does not change their
semantics or the CRD. The security condition above may deserve a note in
the design proposals in `cozystack/community`, which is not a
trigger-map item.
### Release note
```release-note
feat(observability-proxy): add per-application logs and metrics roots under `/apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/{log,metrics}/`, scoped by application identity labels. Data appears for logs and series collected after the application identity changes to Fluent Bit, VMAgent and kube-state-metrics are rolled out.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **New Features**
* Added a tenant-aware observability proxy for logs and metrics, with
namespace and application scoping, access checks, and per-user request
limits.
* Enabled the proxy in the system package bundle and routed console log
and metrics requests through it.
* Added live log streaming and health, readiness, and metrics endpoints.
* **Bug Fixes**
* Invalid or excessive query parameters are rejected; requests with
uncertain application-metric scope are refused rather than returning
incomplete results.
* **Tests**
* Added end-to-end and integration coverage for tenant isolation,
console routing, access controls, and log streaming.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
This clears the golangci-lint findings on main and fixes the errors
several of them pointed at. With the default linter set (errcheck,
govet, ineffassign, staticcheck, unused), main had 197 findings when
this branch started. Rebased on main at 0f169540, the branch leaves 38,
all deprecations that need a design decision rather than a lint fix;
they are listed below.
Most commits are mechanical. They remove unused helpers and constants,
reword error strings, take the staticcheck simplifications, discard
best-effort cleanup errors explicitly, and move to the FieldsV1 and ptr
accessors. A loop in pkg/lineage that always ended in its first pass
becomes a helper, with a regression test for each fallback. These change
behaviour:
- Lost output is now a failure for the commands this PR touches: cozypkg
list, the validation report of validate, push and tap, the push status
line, the tap summary, and check-readiness. The command exits non-zero
and says whether the action it reports on already happened. A closed
pipe ends the process with SIGPIPE, as for other Unix tools.
- backport-batch writes its output before it pushes or publishes. Once a
write to stdout or stderr fails, the run changes nothing more: no push,
no GitHub write, no local branch move, and no new pick. A pick already
in flight still finishes its rollback. The run exits 2, and it catches
SIGPIPE, so a closed pipe is handled the same way. A confirmation
printed after an action is lost only when stderr itself fails, so a lost
confirmation can only be reported by the exit code.
- The SSE server drops a client as soon as a write to it fails. A stream
still open at shutdown no longer panics on a double close, the handler
waits for its pump before returning, and a stream that opens during
shutdown gets 503.
- The kube-ovn plunger now fails its manager when the status server
cannot bind :18080, as a metrics server bind failure does. Before, it
ran without the endpoint and logged nothing.
- A timed-out check-readiness --wait still ends with its verdict, and
the help lists the exit statuses.
The 37 findings left are SA1019 deprecations, kept out of this PR on
purpose:
- GetEventRecorderFor (12, in cmd/*/main.go). The events.k8s.io recorder
has a different interface and needs RBAC for that API group. It changes
what events look like, so it needs its own design.
- client.Apply (12, one more site arrived from main). Moving to
Client.Apply with apply configurations changes how every server-side
apply call is built and how the fake client in tests handles it. It
overlaps the dependency rollup in #4869 and should come after it.
- Result.Requeue (10). Requeue retries with rate-limited backoff,
RequeueAfter with a fixed delay. The interval has to come from whoever
owns each controller.
- scheme.Builder (3, in api/*). Replacing it changes the type of the
exported SchemeBuilder that other repositories import.
- cel.Functions (1, in pkg/reservation). It stays for its late binding,
as the comment next to it explains.
Each behaviour change has tests that failed before the fix. I ran go
test on every changed package (for cmd/backport-batch, the tests around
output and publishing, since the full package takes about ten minutes),
go build ./..., go vet on the changed packages, and a final
golangci-lint run with uncapped output on the rebased tree:
`golangci-lint run --verbose --max-same-issues=0
--max-issues-per-linter=0`, which reports the 38 findings above and
nothing else.
The tests in pkg/lineage, pkg/ovnstatus, pkg/reservation, cmd/cozypkg
and cmd/check-readiness now run in CI: a separate commit adds those
packages to go-unit-tests. Some packages with tests are still in no CI
target: cmd/cozystack-controller, cmd/cozystack-operator,
cmd/observability-proxy, pkg/apiserver, pkg/apis/apps/defaulting,
pkg/apis/apps/install, pkg/apis/apps/presets, pkg/apis/apps/validation,
pkg/apis/core/install, pkg/apis/sdn/install and pkg/version. That
predates this PR.
One known nit: when backport-batch refuses an action after its output
was lost, stderr can show two "output was lost" lines.
### Screenshots
No UI change.
### Downstream repositories
I walked the trigger map against the diff. cozypkg counts as developer
tooling, but only its error paths changed, not a flag, its usage or its
output format, so the development page needs no update. Telemetry metric
names and labels are unchanged.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
chore: cozypkg, check-readiness and backport-batch now fail when their output cannot be written, and backport-batch pushes and publishes nothing once that happens. The kube-ovn plunger fails when its status server cannot start, drops status clients whose writes fail, and no longer panics when it shuts down with a client connected.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Bug Fixes**
* Commands now report output failures instead of silently succeeding.
Backport operations stop before further external changes when required
output cannot be written; readiness checks return a failure status when
output writing fails.
* Package listings, validation reports, push status messages, and tap
summaries now report output errors. Tap commands indicate when changes
were applied before a summary could be written.
* Live event streams handle failed writes and shutdown more reliably,
and status-server startup errors are reported.
* **Documentation**
* Readiness-check help now explains exit statuses and closed-pipe
behavior.
* **Improvements**
* Error messages for several backup and inventory operations now provide
clearer context.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The drain tests now assert that RunThenDrain returns the run's nil
error, the idle-tail test fails on any error other than the stream
ending or being cut, and a test fixture file fails the test when it
cannot be closed.
Removing a temporary directory and closing a response body that has
been read are best-effort: a failure leaves nothing for the caller to
act on and does not change any result, so those errors are discarded
explicitly. The same holds for the stub servers' response writes, whose
failure surfaces on the client side of the test.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A cluster vmselect answers label values with isPartial when a storage
node does not respond, and the list it returns misses that node's
names. Scoping the query to it would drop those pods' series without a
word, which the lookup exists to prevent, so a partial list now fails
the request as any other failed lookup does.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Application roots take two access reviews before any body is read; a
POST to either must answer 503 within the request timeout when
kube-apiserver stalls on them.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The reason given for the second access review held only for the metrics
root. The log root needs it too, because an ownerless pod keeps the
lineage labels its creator set and can claim any application. The
tenant ClusterRole comment also named only the namespace roots, though
the application roots now depend on the same rule.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
An instant query without time had its application lookup sized at the
proxy's clock, but the time was not forwarded, so vmselect evaluated at
its own later clock. With lookups taking most of the request deadline,
the evaluation lands well past the end of the lookup and leaves out pods
first sampled in between. Forward the instant the lookup was sized for.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The string tracking treated a backslash inside backquotes as literal,
but metricsql's scanString lets a backslash escape the closing quote
in all three quote kinds, and a quoted label name keeps that escape
({`\``=""} parses). The proxy ended such a string early, read the rest
of the query as string content, and accepted an escaped
rollup_candlestick after it with the lookup ending too soon.
Inside any quote a backslash now skips the next character, as
scanString does, and the comment no longer claims metricsql refuses
an escaped backquote.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The second latency offset was added to the application lookup only,
so the other readers of the instant query's earliest read still
assumed time minus 30s: the store choice could send a query to the
short-term store up to 30 seconds too late near its retention edge,
the lookback cap compared against a start the lookup no longer used,
and the field no longer meant the earliest instant read.
The earliest read of an instant query now starts from time minus two
latency offsets, the earliest instant vmselect can evaluate it at, and
the lookup uses it as is.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The escape check tracked string literals but not comments: in
metricsql a "#" outside a string comments out the rest of the line, so
a quote inside a comment made the check treat the next line's code as
string content, and rollup\_candlestick there was accepted again. The
bracket and offset scans do not understand comments either.
A "#" outside a string now makes the lookahead unknown as well, which
answers 422 on the application root, rather than teaching every scan
about comments; API clients do not send them.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
vmselect evaluates an instant query whose time is within 30s of its
own clock at that clock minus 30s (queryOffset in prometheus.go), not
at time minus 30s. For a time slightly in the future the evaluation
starts earlier than the proxy assumed: time=now+29s with a 1s step
read from six minutes before, while the lookup started at five
minutes 32 seconds, and pods present only in that gap were missing
from a 200.
The evaluation instant is always within (time-60s, time], so the
lookup of an instant query now starts one more latency offset early.
That needs no assumption about the proxy's clock matching vmselect's.
The store choice on the namespace root is left as it is.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
metricsql unescapes identifiers (unescapeIdent in parser.go), so
rollup\_candlestick(x + y) is rollup_candlestick to vmselect while the
lexical scan saw another name: the lookup ended ten minutes past the
evaluation time and vmselect read nearly thirteen. Any escaped keyword
the scan looks for, offset included, had the same hole. A backslash
outside a string literal now makes the scan unknown, which answers 422
on the application root; escapes inside label values are unaffected.
With rollup_candlestick refused, two steps per "(" stay enough: a
subquery evaluates at most one step past its end plus the alignment of
that end to the step (evalRollupFuncWithSubquery), and the one step a
candlestick over a non-selector would add on top cannot occur.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
metricsql also reads a trailing $__interval as part of an offset's
duration: offset -1h$__interval is -(1h + step). The scan stopped at
the "$" and read one hour, so the lookup missed pods created in the
last step and the query answered 200 without them. Refusing a minus
alone left every other continuation open.
Only what certainly ends the duration token is now accepted after it:
whitespace, a closing bracket or separator, or a binary operator other
than minus; anything else answers 422 on the application root. The
grammar comment no longer claims upper-case offset units are refused,
as offsets are scanned lower-cased like metricsql reads them.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Since the lookup reaches subquery steps past the evaluation time, the
applicationScope comment still described only negative offsets, and a
query refused for a subquery step the scan cannot read got a 422 that
listed every other cause but that one.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The subquery sizing was tested only as "at least two hours", so which
step drives it, the fallback to the query step for an empty subquery
step, and its refusals (an unreadable step, an unclosed bracket, a sum
past the duration range) were not held by any test.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Telling a compound duration from a subtraction by a duration-part
pattern missed forms metricsql accepts after the minus: offset 5m-1.h
is 55 minutes forward and offset 5m-.5h is 25, yet both read as five
minutes back, and the lookup silently missed pods created in between.
A pattern that followed scanSingleDuration exactly would still be a
second copy of the grammar to keep in step, so any minus right after
an offset's duration is refused again; offset 1h-1 and offset 5m-y now
answer 422 as well, and metricsql cannot parse the latter anyway.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The compound-offset refusal took any minus right after an offset's
duration for a compound, so x offset 1h-1 or x offset 5m-y answered
422. metricsql continues a duration token only with a minus followed
by a duration part (scanSingleDuration), so only that form is refused
now; anything else is the subtraction it is.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
vmselect evaluates a subquery up to its end plus one subquery step,
rounded up to the step (evalRollupFuncWithSubquery in
app/vmselect/promql/eval.go), and range_* transforms (setLastValues)
and keep_next_value carry those later values back onto earlier points.
A series that starts after the evaluation time, such as a pod created
within that reach, can therefore appear in the answer, which the
earlier claim that only rollup_candlestick reads ahead missed. The
lookup stopped at the evaluation time and left such a pod out of a 200.
The lookahead now adds two steps per subquery level, counting every
"(" as a possible implicit subquery and using the largest step seen,
as the lookback already does. Overreaching only widens the lookup over
the application's own pods.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
vmselect evaluates rollup_candlestick one step ahead of the evaluation
time, with no offset in the query: evalRollupFuncWithoutAt adds the
step in effect to the window's start and end (app/vmselect/promql/
eval.go in v1.136.0-cluster, unchanged in v1.139.0). The lookahead
scan returned zero for it, so the lookup stopped at the evaluation time
and pods created during that step were left out of a 200.
The step in effect is the subquery's when the call is nested, so
rather than size it the application root answers 422. No other
function reads ahead: the subquery evaluation past its end in
evalRollupFuncWithSubquery is outside the outer rollup's window.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The offset scan read a duration up to its first sign and ignored the
rest. metricsql lexes 1h-30m as one duration and DurationValue makes
every part after the first negative one subtract, so offset -1h-30m
shifts samples 90 minutes forward while the scan covered 60, and
offset 1h-2h shifts them an hour forward while the scan counted it as
looking back. The pod and PVC lookup then missed pods created in the
gap and the query answered 200 without them.
A duration followed directly by a minus now makes the scan unknown,
which answers 422 on the application root. A plus or a spaced sign is
an operator after the offset in metricsql, not part of it, and is
still read.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The refusal named only the @ modifier and non-literal offsets, but the
same answer covers an @ inside a label value and WITH with a negative
offset, so a client could not tell why such a query was refused.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The series-limit refusal of a lookup reached the client with all of
cluster vmselect's wrapping, and that context can name vmstorage
nodes, the leak the generic 502 had just closed for other errors. The
client now gets the message from the limit on.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
A store error other than the series limit was built into the lookup's
502 message, and that text can name vmstorage nodes and addresses.
The handler happened to replace the message before it reached the
client, but nothing held it there. The text now goes to the log only.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The lookahead scan let @ start() and @ end() through and the 422 named
them as allowed, but the lookback scan cannot bound any @, so every @
query was refused anyway, with a message pointing at forms that would
not pass. Both scans now treat any @ as unknown and the message says
so. The lookback-cap test now exercises the cap with a window just past
it rather than an @ query.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The lookup now passes a 504 from the store through, where it used to
answer 502, but only the 504 read from a 422 body was tested.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
vmselect answers 422 to every error that carries no status of its own,
its query timeout and storage failures included, so passing every
lookup 422 through told the client to narrow a window when the store
was failing, and bypassed the 504 for a missed deadline.
Only vmselect's series-limit message stays a 422, with the message
taken from its JSON answer; its timeout becomes 504 and anything else
502. The request deadline is checked before the store's status.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
With the lookback refusal, the 24h default turned common application
queries into 422: a day's increase, and month-long panels at the step
a dashboard picks for them. The lookback estimate is a loose upper
bound, made for the store choice, and counts every "(" as an implicit
subquery, so it far exceeds what such queries read.
The lookback is only the refusal threshold, since the lookup itself
spans the window the query reads, so its default now matches
--metrics-max-range. The chart passes it from values, so an operator
can tune it.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
vmselect answers 422 when a label lookup exceeds its own series
limits. The application root turned that into a 502, which reads as a
broken store, while the client could have fixed it by narrowing the
window. The 422 and vmselect's message now reach the client.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The pod and PVC lookup never started earlier than --metrics-app-lookback
before the window. A query that read further back, such as rate(x[7d])
with the 24h default, or whose lookback is unknown (an @ modifier, WITH,
a step-multiple window), answered 200 without the series of pods that
existed only before the lookup start.
Such queries now answer 422 on the application root, matching how a
query whose latest sample cannot be covered is handled; the namespace
root has no lookup and still serves them.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Only a deadline hit before the lookup's response headers was reported
as 504. When the store sent its headers and then stalled, the deadline
expired while the body was decoded, and the decode error came back as
a generic 502, so a slow store looked like a broken one.
The request deadline is now checked whichever step of the lookup it
cut, and both cases are covered by a test.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
vmselect rounds a range query's end up to a whole step before
evaluating it, so the last point can read samples up to one step past
the end the client sent. The pod and PVC lookup stopped at that end
and could miss a pod created within the last step.
The lookup end of a range query now adds one step.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
When the pod or PVC lookup failed, its branch was dropped and the
query still answered 200, without the cAdvisor and volume series of
the application, so a store error showed up as an empty graph. That
is the same silently incomplete answer that a lookup over its cap is
refused for.
A failed lookup now fails the request: 504 when the store missed the
deadline, 502 otherwise.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Reviews was inserted between ResourceAttributes and its doc comment,
so go doc showed the ResourceAttributes text on Reviews, and that text
still called it the single review a route needs, which application
roots no longer are.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The pod and PVC lookup of the application metrics root ended at the
evaluation time, but a negative offset reads samples after it. An
instant query evaluated two hours ago with "offset -1h" reads samples
from one hour ago, so a pod created in between was missing from the
scope and its series were dropped while the request succeeded.
The lookup now ends at the latest instant the query reads, its window
end plus the sum of its negative offsets. A query whose latest sample
cannot be bounded, an @ modifier other than start() or end() or an
offset that is not a plain duration, is refused with 422 on the
application root; the namespace root has no lookup and still serves
it.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The application roots asked only for the application subresource. Their
pod and PVC branches select by namespace and name across the whole
lookup window, so a caller granted only application "a" received the
older series of application "b" whenever "b" had used one of the same
pod or PVC names. The lineage labels the scope rests on can also be set
by whoever creates an unowned pod.
Until the scope carries application identity on every branch, the
application roots also require read access to the namespace root, the
same review the namespace root asks. Every tenant role that reads
applications already holds it, so no current grant loses access; a
grant narrowed by resourceNames alone is refused instead of trusted.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The store cuts a lookup at the cap plus one value. Dropping empty
values first could bring such a cut list back under the cap and serve
it as complete, hiding the application's other pods without the 422
that asks for a shorter range.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
An empty name in a lookup answer would turn the pod branch into a
pattern that matches every series without a pod label, widening the
application scope to the namespace; nothing pinned the guard against
it. Malformed answers must only drop their branch, and the lookup must
start at the query's own lookback, or at the configured limit when that
lookback is unknown.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Graphs of one application need its series without the caller knowing
which pods and volumes it ran on. Exporter series carry the identity
the scrape class stamps, but cAdvisor and kubelet volume series do not,
so the scope ORs three branches: the identity, and the application's
pods and PVCs as kube-state-metrics labelled them over the window. Each
branch also carries the namespace, so the scope is never wider than the
namespace root.
The lookup reaches back by the query's own lookback, bounded by a flag,
so pods replaced inside an expression's range still count. A failed
lookup only drops its branch. A list over its cap answers 422 instead
of being truncated, which would hide series without saying so.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
A viewer of one application gets that application's lines without
writing a namespace query and filtering by hand, and a grant narrowed to
one application by resourceNames is honoured on every logs endpoint,
tail and the pickers included. The namespace filter stays and the
application's identity is ANDed next to it as a second stream filter,
so the result can only be narrower than the namespace root.
The plural is mapped to its kind only after the caller is authorized,
so an anonymous client cannot probe which applications exist, and the
proxy is not ready until the ApplicationDefinitions cache is.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Paths and RBAC name an application by its plural, while the lineage
labels that scope its logs and series carry the kind. The mapping comes
from ApplicationDefinitions, kept in one place so both application roots
scope by the same kind. A plural declared for two kinds is refused
rather than resolved by cache order.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The metrics handler shares the bounded review calls with the logs
handler; a POST to it with an unread body must answer 503 as well
rather than wait on a stalled kube-apiserver.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
staticcheck flags the negated disjunctions; naming the duration
characters and spelling out both word boundaries reads the same way the
rule is stated.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Only the upper half of the refused 1970..2262 range had a test, so
the lower check could be dropped unnoticed. vmselect clamps a time
before 1970 to 0, which would answer for a window other than the one
asked for.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Times went to vmselect as "%d.%03d" unix seconds. vmselect reads a
number without "-" as a unix time whose unit it guesses from its size,
so a time in the first 24.8 days of 1970 arrived as a thousand times
as many seconds: a window the proxy checked as 8.8 days and 30 points
reached vmselect as 2001..2025 and 30000 points, and the label and
series APIs, which have no points check, scanned the whole retention.
Times past 2038 were read as microseconds.
Times now go out as RFC3339 with milliseconds, which vmselect parses
as a date and never as a guessed unit, and times outside 1970..2262,
which vmselect cannot hold as nanoseconds, are refused. A test repeats
vmselect's parsing of the exact strings the proxy sends.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The metrics slow-body and stalled-client tests probed the slot with
requests of their own, which competed for the one slot under test and
made them fail at random, under one shuffle seed reliably. They now
read the limiter's slot count, as the logs tests do; 300 runs of the
test and 20 shuffled runs of the package pass.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The metrics concurrency test only checked that a Retry-After was
present, while the rate limit test pins its value, so a change to the
promised hint passed. It now expects the literal value.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
vmselect reads step and timeout as float seconds multiplied by 1000
and truncated, so a step the proxy checked as 1001ms and sent as
"1.001" arrived as 1000ms. The points limit was then checked against
a coarser step than vmselect used: a limit of 11000 let 11010 points
through, and many steps between 1ms and 100000s were affected.
Durations are now written so that the truncation lands on the
milliseconds the limits were checked with: the shortest form when it
reads back exactly, otherwise with half a millisecond added. Step and
timeout share the formatter, and a sweep test mirrors vmselect's
parsing.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The metrics root has its own admission, so the hint added to the logs
root did not reach it, and a console graph polling into the limit had
no back-off hint.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
vmselect's sample limit counts raw samples read, not the work of a
rollup that revisits each sample across overlapping windows, so for
that work the forwarded timeout is the limit that applies.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The comment claimed the limits bound everything a request makes
vmselect do, but windows and offsets inside the expression pass
through unchecked: they are bounded by the store's retention and by
vmselect's own series and sample limits, not by the proxy. A reader
taking the comment at its word would rely on a cost cap that does not
exist.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>