Commit Graph
24 Commits
Author SHA1 Message Date
ZverGuy 336961d7cd docs(observability-proxy): put the store-call multiplier in the flag help
The note that an application metrics request makes three store calls
sat in a source comment, where an operator reading --help never sees
it. It is now part of the --rate-limit-per-user help text.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 85fb98dd2c fix(observability-proxy): stop refusing ordinary app queries on lookback
With the lookback refusal, the 24h default turned common application
queries into 422: a day's increase, and month-long panels at the step
a dashboard picks for them. The lookback estimate is a loose upper
bound, made for the store choice, and counts every "(" as an implicit
subquery, so it far exceeds what such queries read.

The lookback is only the refusal threshold, since the lookup itself
spans the window the query reads, so its default now matches
--metrics-max-range. The chart passes it from values, so an operator
can tune it.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 86dddb447a docs(observability-proxy): note the store calls behind one app request
The per-user rate counts client requests, but an application metrics
request makes three store calls, so an operator sizing the rate
against store capacity needs to know the multiplier.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 84d5b4ff0a fix(observability-proxy): require a positive application lookback
Every query reads some time before its window, at least the implicit
rollup window, so with the lookback refusal a zero
--metrics-app-lookback would turn every application metrics query
into a 422. It is now refused at startup instead.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 84665a42a7 fix(observability-proxy): refuse app queries that read past the lookup
The pod and PVC lookup never started earlier than --metrics-app-lookback
before the window. A query that read further back, such as rate(x[7d])
with the 24h default, or whose lookback is unknown (an @ modifier, WITH,
a step-multiple window), answered 200 without the series of pods that
existed only before the lookup start.

Such queries now answer 422 on the application root, matching how a
query whose latest sample cannot be covered is handled; the namespace
root has no lookup and still serves them.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy dcdf0adab3 feat(observability-proxy): serve the application metrics root
Graphs of one application need its series without the caller knowing
which pods and volumes it ran on. Exporter series carry the identity
the scrape class stamps, but cAdvisor and kubelet volume series do not,
so the scope ORs three branches: the identity, and the application's
pods and PVCs as kube-state-metrics labelled them over the window. Each
branch also carries the namespace, so the scope is never wider than the
namespace root.

The lookup reaches back by the query's own lookback, bounded by a flag,
so pods replaced inside an expression's range still count. A failed
lookup only drops its branch. A list over its cap answers 422 instead
of being truncated, which would hide series without saying so.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 2f600d5354 feat(observability-proxy): serve the application logs root
A viewer of one application gets that application's lines without
writing a namespace query and filtering by hand, and a grant narrowed to
one application by resourceNames is honoured on every logs endpoint,
tail and the pickers included. The namespace filter stays and the
application's identity is ANDed next to it as a second stream filter,
so the result can only be narrower than the namespace root.

The plural is mapped to its kind only after the caller is authorized,
so an anonymous client cannot probe which applications exist, and the
proxy is not ready until the ApplicationDefinitions cache is.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy b3288be903 test(observability-proxy): cite only sources the repository has
The flag defaults test named a design document that is not in the
repository as the origin of the cost limits; the limits stand on their
own as flag defaults.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:06 +07:00
ZverGuy 3e2472280e fix(observability-proxy): keep the vmselect timeout below the request's
An equal vmselect timeout still lost the race: the proxy's deadline
starts before the request reaches vmselect, so the proxy cut first
and answered 504 for a store within its time. The vmselect timeout
must now be strictly below a set request timeout, the flag help says
so, and tests pin the accepted side too, including no request timeout.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 03:56:18 +07:00
ZverGuy 0c5d1cd93f fix(observability-proxy): keep the vmselect timeout within the request's
The timeout forwarded to vmselect could be set longer than the proxy's
own request timeout. The proxy then cut a query vmselect was still
allowed to run and answered 504 for a store that was within its time.
Such a combination now stops the proxy at startup.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 03:56:18 +07:00
ZverGuy 81cd278f63 test(observability-proxy): cover metrics concurrency and flag overrides
The per-user concurrency cap on the metrics root had no test, so
dropping it from the handler went unnoticed. The new test holds one
request in the upstream, expects the same user's second one to be
refused and another user's to go through. Overrides of the metrics
flags were only checked for their defaults and refusals.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 03:56:18 +07:00
ZverGuy cd3d889ee8 feat(observability-proxy): add metrics store and cost limit flags
The store names and the short-term retention mirror metricsStorages in
the monitoring package, which an operator can rename or resize; they
are flags so the chart can follow such a change without a rebuild. The
cost limit defaults are the ones the metrics proposal settles on: 30d
range, 11000 points per series, 30s timeout.

Store names become part of a Service host name, so they are checked
as DNS labels at startup.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 03:56:18 +07:00
ZverGuy 59cbf4b28e docs(observability-proxy): say what the request timeout bounds
The flag now also bounds the body read and the token and access
reviews, and 0 lifts all of those at once; an operator raising it for
slow store queries needs to know that. The store comment named a
design document the repository does not have.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 03:53:24 +07:00
ZverGuy d2ef5e031f docs(observability-proxy): describe the proxy as it stands
The package and serve comments described a delivery plan rather than
the code: the command named metrics it does not serve, and serve gave a
plan as the reason to refuse unknown roots before authenticating.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 03:53:24 +07:00
ZverGuy 9d9f5fcb6c docs(observability-proxy): name the source of the logs bucket default
The default of 11000 buckets was justified by a metrics proxy that
does not exist on this branch. The number is Prometheus's cap on
points per series in a range query, which the metrics root reuses.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy b1108e836a fix(observability-proxy): refuse to resolve a store before the cache syncs
An absent namespace resolves to the root store, and before the
informer's first list every namespace is absent, so a request served
in that window read a labeled namespace's logs from the wrong store.
Readiness keeps the Service from routing there, but a client that
addresses the pod directly is not stopped by it.

The resolver now fails while the cache is unsynced, which answers
503 like any other resolution error.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy cd04c3e6c4 fix(observability-proxy): count the aligned buckets in the logs step
vlselect widens the window to step boundaries and returns a
zero-filled bucket for each one, so a start off a boundary yields one
bucket more than the span divided by the step. Sizing the step from
the unaligned span let hits and stats_query_range return MaxBuckets+1
buckets.

The minimum step now spans MaxBuckets-1 steps, and the smallest
accepted --logs-max-buckets rises to 4 so that this step still fits
the third-of-range cap that keeps alignment inside the window.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy 2d8de08724 fix(observability-proxy): refuse positional arguments
The flag parser stops at the first argument that is not a flag, so
a stray word in the container args made every flag after it fall back
to its default without an error, the root tenant included.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy 2e3edcf30d fix(observability-proxy): refuse a non-finite per-user rate
NaN fails the negative check, as every comparison does, and +Inf
refills a bucket instantly, so either value switched the per-user rate
limit off while the flag looked set. Both now stop the proxy at
startup.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy a56ff5050f test(observability-proxy): accept the smallest satisfiable step cap
Startup refuses --logs-max-buckets below 3 and --logs-max-range below
3s, but only the refusals were tested, so moving either bound up by
one went unnoticed. Pin the smallest accepted pair.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
ZverGuy ca7f283b6a fix(observability-proxy): require a range and bucket count the step cap fits
The step is capped at a third of the maximum range so that vlselect's
alignment to whole steps cannot widen the window past that range. With
fewer than three buckets, or a range under three seconds, the minimum
step exceeds that cap and the bound no longer holds, so such settings
are refused at startup.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
ZverGuy 8eef2fa5d2 fix(observability-proxy): reject negative limits at startup
Zero is the documented way to disable a limit; a negative value
disabled it too, silently. Refuse it so a typo in the chart cannot turn
off a per-user or total cap unnoticed.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
ZverGuy 1c345da367 fix(observability-proxy): reject negative timeouts at startup
A negative duration silently disables the bound it configures, so a
typo in the chart would turn off the request deadline or the tail idle
close without any sign. Refuse it when the flags are parsed.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
ZverGuy 8b0ac342b5 feat(observability-proxy): add the observability-proxy command
The deploy unit builds an image and chart around this binary, so its
flags follow the agreed contract: listen and probe addresses, cluster
domain, root tenant and auth cache TTL, plus the logs limits. Defaults
are chosen so a stock install is safe without tuning; readiness waits
for the namespace cache because store resolution reads it on every
request. On shutdown open tails are cut after a grace period so their
clients reconnect with since to another replica.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00