Commit Graph
21 Commits
Author SHA1 Message Date
ZverGuy 6fe3e5f6ff fix(observability-proxy): refuse a partial application lookup
A cluster vmselect answers label values with isPartial when a storage
node does not respond, and the list it returns misses that node's
names. Scoping the query to it would drop those pods' series without a
word, which the lookup exists to prevent, so a partial list now fails
the request as any other failed lookup does.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy ac5a3c88c4 fix(observability-proxy): put the latency window in the instant start
The second latency offset was added to the application lookup only,
so the other readers of the instant query's earliest read still
assumed time minus 30s: the store choice could send a query to the
short-term store up to 30 seconds too late near its retention edge,
the lookback cap compared against a start the lookup no longer used,
and the field no longer meant the earliest instant read.

The earliest read of an instant query now starts from time minus two
latency offsets, the earliest instant vmselect can evaluate it at, and
the lookup uses it as is.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy cf45a23bf3 fix(observability-proxy): refuse comments outside strings on the app root
The escape check tracked string literals but not comments: in
metricsql a "#" outside a string comments out the rest of the line, so
a quote inside a comment made the check treat the next line's code as
string content, and rollup\_candlestick there was accepted again. The
bracket and offset scans do not understand comments either.

A "#" outside a string now makes the lookahead unknown as well, which
answers 422 on the application root, rather than teaching every scan
about comments; API clients do not send them.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy a54760fd9c fix(observability-proxy): start the instant app lookup a latency early
vmselect evaluates an instant query whose time is within 30s of its
own clock at that clock minus 30s (queryOffset in prometheus.go), not
at time minus 30s. For a time slightly in the future the evaluation
starts earlier than the proxy assumed: time=now+29s with a 1s step
read from six minutes before, while the lookup started at five
minutes 32 seconds, and pods present only in that gap were missing
from a 200.

The evaluation instant is always within (time-60s, time], so the
lookup of an instant query now starts one more latency offset early.
That needs no assumption about the proxy's clock matching vmselect's.
The store choice on the namespace root is left as it is.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 3198de893a fix(observability-proxy): refuse escapes outside strings on the app root
metricsql unescapes identifiers (unescapeIdent in parser.go), so
rollup\_candlestick(x + y) is rollup_candlestick to vmselect while the
lexical scan saw another name: the lookup ended ten minutes past the
evaluation time and vmselect read nearly thirteen. Any escaped keyword
the scan looks for, offset included, had the same hole. A backslash
outside a string literal now makes the scan unknown, which answers 422
on the application root; escapes inside label values are unaffected.

With rollup_candlestick refused, two steps per "(" stay enough: a
subquery evaluates at most one step past its end plus the alignment of
that end to the step (evalRollupFuncWithSubquery), and the one step a
candlestick over a non-selector would add on top cannot occur.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 054fac6ab0 docs(observability-proxy): name subquery steps in the app lookup texts
Since the lookup reaches subquery steps past the evaluation time, the
applicationScope comment still described only negative offsets, and a
query refused for a subquery step the scan cannot read got a 422 that
listed every other cause but that one.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy cb35399ee7 fix(observability-proxy): refuse rollup_candlestick on the app root
vmselect evaluates rollup_candlestick one step ahead of the evaluation
time, with no offset in the query: evalRollupFuncWithoutAt adds the
step in effect to the window's start and end (app/vmselect/promql/
eval.go in v1.136.0-cluster, unchanged in v1.139.0). The lookahead
scan returned zero for it, so the lookup stopped at the evaluation time
and pods created during that step were left out of a 200.

The step in effect is the subquery's when the call is nested, so
rather than size it the application root answers 422. No other
function reads ahead: the subquery evaluation past its end in
evalRollupFuncWithSubquery is outside the outer rollup's window.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 486f13dce6 docs(observability-proxy): name every query form the app 422 refuses
The refusal named only the @ modifier and non-literal offsets, but the
same answer covers an @ inside a label value and WITH with a negative
offset, so a client could not tell why such a query was refused.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 5fc9e5a0b9 fix(observability-proxy): pass on only the limit text of a lookup 422
The series-limit refusal of a lookup reached the client with all of
cluster vmselect's wrapping, and that context can name vmstorage
nodes, the leak the generic 502 had just closed for other errors. The
client now gets the message from the limit on.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 41e61cb288 fix(observability-proxy): keep store error text out of lookup answers
A store error other than the series limit was built into the lookup's
502 message, and that text can name vmstorage nodes and addresses.
The handler happened to replace the message before it reached the
client, but nothing held it there. The text now goes to the log only.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 0bb98ced05 fix(observability-proxy): say that every @ is refused on app roots
The lookahead scan let @ start() and @ end() through and the 422 named
them as allowed, but the lookback scan cannot bound any @, so every @
query was refused anyway, with a message pointing at forms that would
not pass. Both scans now treat any @ as unknown and the message says
so. The lookback-cap test now exercises the cap with a window just past
it rather than an @ query.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy caa4e8d0e3 fix(observability-proxy): keep only the series-limit 422 of a lookup
vmselect answers 422 to every error that carries no status of its own,
its query timeout and storage failures included, so passing every
lookup 422 through told the client to narrow a window when the store
was failing, and bypassed the 504 for a missed deadline.

Only vmselect's series-limit message stays a 422, with the message
taken from its JSON answer; its timeout becomes 504 and anything else
502. The request deadline is checked before the store's status.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 1a3e1827ef fix(observability-proxy): pass a store 422 from the app lookup through
vmselect answers 422 when a label lookup exceeds its own series
limits. The application root turned that into a 502, which reads as a
broken store, while the client could have fixed it by narrowing the
window. The 422 and vmselect's message now reach the client.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 84665a42a7 fix(observability-proxy): refuse app queries that read past the lookup
The pod and PVC lookup never started earlier than --metrics-app-lookback
before the window. A query that read further back, such as rate(x[7d])
with the 24h default, or whose lookback is unknown (an @ modifier, WITH,
a step-multiple window), answered 200 without the series of pods that
existed only before the lookup start.

Such queries now answer 422 on the application root, matching how a
query whose latest sample cannot be covered is handled; the namespace
root has no lookup and still serves them.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy bc823b704e fix(observability-proxy): answer 504 to a lookup that misses the deadline
Only a deadline hit before the lookup's response headers was reported
as 504. When the store sent its headers and then stalled, the deadline
expired while the body was decoded, and the decode error came back as
a generic 502, so a slow store looked like a broken one.

The request deadline is now checked whichever step of the lookup it
cut, and both cases are covered by a test.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 1a3517fa81 fix(observability-proxy): fail an application query whose lookup failed
When the pod or PVC lookup failed, its branch was dropped and the
query still answered 200, without the cAdvisor and volume series of
the application, so a store error showed up as an empty graph. That
is the same silently incomplete answer that a lookup over its cap is
refused for.

A failed lookup now fails the request: 504 when the store missed the
deadline, 502 otherwise.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 07f4f187ce fix(observability-proxy): cover forward-shifted samples in the app lookup
The pod and PVC lookup of the application metrics root ended at the
evaluation time, but a negative offset reads samples after it. An
instant query evaluated two hours ago with "offset -1h" reads samples
from one hour ago, so a pod created in between was missing from the
scope and its series were dropped while the request succeeded.

The lookup now ends at the latest instant the query reads, its window
end plus the sum of its negative offsets. A query whose latest sample
cannot be bounded, an @ modifier other than start() or end() or an
offset that is not a plain duration, is refused with 422 on the
application root; the namespace root has no lookup and still serves
it.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy c4593d4c3e fix(observability-proxy): require namespace access on application roots
The application roots asked only for the application subresource. Their
pod and PVC branches select by namespace and name across the whole
lookup window, so a caller granted only application "a" received the
older series of application "b" whenever "b" had used one of the same
pod or PVC names. The lineage labels the scope rests on can also be set
by whoever creates an unowned pod.

Until the scope carries application identity on every branch, the
application roots also require read access to the namespace root, the
same review the namespace root asks. Every tenant role that reads
applications already holds it, so no current grant loses access; a
grant narrowed by resourceNames alone is refused instead of trusted.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy dd7116ea45 fix(observability-proxy): check the lookup cap before dropping empty names
The store cuts a lookup at the cap plus one value. Dropping empty
values first could bring such a cut list back under the cap and serve
it as complete, hiding the application's other pods without the 422
that asks for a shorter range.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy e6697ea4f1 test(observability-proxy): cover malformed pod lookups and the lookup window
An empty name in a lookup answer would turn the pod branch into a
pattern that matches every series without a pod label, widening the
application scope to the namespace; nothing pinned the guard against
it. Malformed answers must only drop their branch, and the lookup must
start at the query's own lookback, or at the configured limit when that
lookback is unknown.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy dcdf0adab3 feat(observability-proxy): serve the application metrics root
Graphs of one application need its series without the caller knowing
which pods and volumes it ran on. Exporter series carry the identity
the scrape class stamps, but cAdvisor and kubelet volume series do not,
so the scope ORs three branches: the identity, and the application's
pods and PVCs as kube-state-metrics labelled them over the window. Each
branch also carries the namespace, so the scope is never wider than the
namespace root.

The lookup reaches back by the query's own lookback, bounded by a flag,
so pods replaced inside an expression's range still count. A failed
lookup only drops its branch. A list over its cap answers 422 instead
of being truncated, which would hide series without saying so.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00