mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
feat(observability-proxy): serve per-application logs and metrics (#5220)
## What this PR does Depends on #5219. This PR is stacked on that branch; the description covers only what it adds on top. Adds the per-application roots to the observability proxy, as described in the design proposals [Tenant application logs](https://github.com/cozystack/community/pull/95) and [Tenant resource-consumption metrics](https://github.com/cozystack/community/pull/94): ```text /apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/log/select/logsql/<endpoint> /apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/metrics/api/v1/<endpoint> ``` This lets the console show the logs and resource consumption of one application rather than of the whole namespace. What is added: - Plural-to-Kind mapping from an informer over `ApplicationDefinitions`. The Kind is looked up only after the `SubjectAccessReview` (`get <plural>/log` or `get <plural>/metrics`), so an unauthenticated caller cannot probe which plurals exist. An unknown plural answers `404`; an unsynced cache or a plural declared by two kinds answers `503`. - Logs: a second `extra_stream_filters` value with the application Kind and name, on top of the unchanged namespace filter. VictoriaLogs ANDs the two, on every endpoint including `tail`. - Metrics: the forced `extra_filters[]` becomes an OR of three branches, each still bound to the namespace: 1. series carrying the application identity labels; 2. series of the application's pods, found through `kube_pod_labels`; 3. series of the application's PVCs, found through `kube_persistentvolumeclaim_labels`. - The pod and PVC lookups cap the number of names (`--metrics-app-max-pods`, `--metrics-app-max-pvcs`); over the cap answers `422`. A failed pod or PVC lookup fails the request (`502`, or `504` on the deadline) rather than answer without the application's series, and so does a lookup that vmselect marks `isPartial`, since a storage node that did not answer leaves names out (a partial list already over the cap still answers `422`); vmselect's series-limit refusal of a lookup is passed through as `422`. A query that reads further back than `--metrics-app-lookback` (default 720h, chart value `observabilityProxy.metrics.appLookback`), or by an unknown amount, and a query whose samples lie at a time the lookup cannot cover answer `422` on the application root; this includes every query with an `@` modifier, `WITH` with a negative offset, and offsets that are not plain durations. The namespace root still serves them. `rollup_candlestick`, a backslash or a `#` comment outside a string, an offset directly followed by a minus (compound durations such as `-1h-30m` or `1h-2h`), and a subquery step that is not a plain duration are refused as well. The lookup reaches past the evaluation time by any negative offset and by two steps per subquery level. - Application routes require read access to the namespace root as well as the application subresource; a grant narrowed with `resourceNames` alone answers `403`. - The proxy's ClusterRole gains `list`/`watch` on `cozystack.io/applicationdefinitions`. No tenant RBAC change is needed: the `view` role already allows `get` on every `apps.cozystack.io` resource and subresource. - `/readyz` now also waits for the `ApplicationDefinitions` cache, so a missing CRD or RBAC rule keeps the whole proxy unready, namespace roots included. #### Data comes from the identity labels The application roots find data only through identity labels added by three PRs that are now in main: - #5215 `feat(monitoring-agents)`: export PVC labels from kube-state-metrics (PVC branch). - #5216 `feat(monitoring)`: stamp application identity on scraped series (identity branch). - #5217 `feat(monitoring-agents)`: keep application identity in container log streams (logs root). Merge after #5219. The pod branch does not depend on them, because `kube_pod_labels` already exports all pod labels. Logs and series written before those changes rolled out carry no identity and are reachable only through the namespace root. #### Security condition: a convenience filter, not an isolation boundary The application roots narrow the namespace roots; they do not isolate applications from each other. Each application route therefore requires two grants: the application subresource (`get <plural>/log` or `get <plural>/metrics`) and read access to the namespace root (`list cozystack.io/logs` or `cozystack.io/metrics` in the namespace). A grant narrowed with `resourceNames` alone is refused. Every tenant role that can read applications already has both. The application scope cannot isolate yet because: 1. The lineage webhook keeps `apps.cozystack.io/application.*` labels that a user sets on a pod or PVC that has no application owner, so such an object can claim another application's identity. 2. The webhook's `objectSelector` skips any object created with `internal.cozystack.io/managed-by-cozystack`, pods included, so forged labels on such an object are never examined. 3. The pod and PVC branches match by name over the lookup window, so an object that reuses the name of another application's recently deleted pod or PVC pulls that object's older series into scope. The proposed fix (a separate change to the lineage webhook) is to strip the identity labels when no owner resolves, and to stop skipping CREATE requests for objects carrying the managed-by label. #### Other known gaps - PVCs created from a StatefulSet's `volumeClaimTemplates` usually get no lineage labels, so their volume series are not in the application scope. - ingress-nginx series intentionally carry no application identity, so they are reachable only through the namespace root. - A request for an application that does not exist answers `200` with an empty result rather than `404`: the proxy checks authorization, not existence. Nothing leaks; whether a `404` is wanted is open for review. #### How it was tested - `go test -race` and `go vet` for the proxy packages, including a narrowed-grant test (access to application `a` only): every application path for `b` answers `403` and no request reaches the store. Name escaping in the pod and PVC branches is covered for regex metacharacters and quotes. - helm-unittest for `observability-proxy` (new RBAC rule). - Verified on a test cluster, with the three identity PRs above applied: the application logs root of a Redis instance returned only that instance's lines and none from a neighbouring pod in the same namespace; its application metrics root returned only the Redis pods' series. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Downstream: no trigger matched. Candidate: none affected, because the diff on top of #5219 is confined to the proxy's Go code and its own chart RBAC; it reads `ApplicationDefinitions` but does not change their semantics or the CRD. The security condition above may deserve a note in the design proposals in `cozystack/community`, which is not a trigger-map item. ### Release note ```release-note feat(observability-proxy): add per-application logs and metrics roots under `/apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/{log,metrics}/`, scoped by application identity labels. Data appears for logs and series collected after the application identity changes to Fluent Bit, VMAgent and kube-state-metrics are rolled out. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added a tenant-aware observability proxy for logs and metrics, with namespace and application scoping, access checks, and per-user request limits. * Enabled the proxy in the system package bundle and routed console log and metrics requests through it. * Added live log streaming and health, readiness, and metrics endpoints. * **Bug Fixes** * Invalid or excessive query parameters are rejected; requests with uncertain application-metric scope are refused rather than returning incomplete results. * **Tests** * Added end-to-end and integration coverage for tenant isolation, console routing, access controls, and log streaming. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
22 files changed
+1678
-50
No files matched your search
@@ -39,6 +39,8 @@ import (
|
||||
"github.com/prometheus/client_golang/prometheus/collectors"
|
||||
"github.com/prometheus/client_golang/prometheus/promhttp"
|
||||
"k8s.io/apimachinery/pkg/util/validation"
|
||||
"k8s.io/client-go/dynamic"
|
||||
"k8s.io/client-go/dynamic/dynamicinformer"
|
||||
"k8s.io/client-go/informers"
|
||||
"k8s.io/client-go/kubernetes"
|
||||
"k8s.io/client-go/tools/cache"
|
||||
@@ -92,8 +94,15 @@ func parseFlags(args []string) (options, error) {
|
||||
fs.DurationVar(&o.metrics.Limits.MaxRange, "metrics-max-range", 30*24*time.Hour, "longest [start, end] and step a metrics request may use")
|
||||
fs.IntVar(&o.metrics.Limits.MaxPoints, "metrics-max-points", 11000, "most points per series a range query may return")
|
||||
fs.DurationVar(&o.metrics.Limits.MaxTimeout, "metrics-max-timeout", 30*time.Second, "timeout forwarded to vmselect; a longer client timeout is clamped; must be below --request-timeout when that is set")
|
||||
// Only the refusal threshold: the lookup itself spans the window the
|
||||
// query reads, and that estimate is a loose upper bound (every "("
|
||||
// counts as an implicit subquery), so the default is long enough to
|
||||
// serve month-long panels at a dashboard step.
|
||||
fs.DurationVar(&o.metrics.App.Lookback, "metrics-app-lookback", 30*24*time.Hour, "how far before the window an application metrics query may read; one reading further back, or by an unknown amount, answers 422")
|
||||
fs.IntVar(&o.metrics.App.MaxPods, "metrics-app-max-pods", 500, "most pods of one application in the lookup window; more answer 422")
|
||||
fs.IntVar(&o.metrics.App.MaxPVCs, "metrics-app-max-pvcs", 500, "most PVCs of one application in the lookup window; more answer 422")
|
||||
fs.IntVar(&o.limiter.MaxConcurrentPerUser, "max-concurrent-per-user", 10, "concurrent non-tail requests per user; 0 disables")
|
||||
fs.Float64Var(&o.limiter.RatePerUser, "rate-limit-per-user", 10, "sustained requests per second per user; 0 disables")
|
||||
fs.Float64Var(&o.limiter.RatePerUser, "rate-limit-per-user", 10, "sustained client requests per second per user; an application metrics request makes three store calls (pod lookup, PVC lookup, query); 0 disables")
|
||||
fs.IntVar(&o.limiter.RateBurst, "rate-limit-burst", 20, "request burst per user")
|
||||
fs.IntVar(&o.limiter.MaxTailsPerUser, "max-tails-per-user", 5, "open tails per user; 0 disables")
|
||||
fs.IntVar(&o.limiter.MaxTails, "max-tails", 200, "open tails in total; 0 disables")
|
||||
@@ -132,6 +141,8 @@ func parseFlags(args []string) (options, error) {
|
||||
return o, errors.New("--metrics-shortterm-retention, --metrics-max-range and --metrics-max-timeout must be positive")
|
||||
case o.metrics.Limits.MaxPoints < 2:
|
||||
return o, errors.New("--metrics-max-points must be at least 2")
|
||||
case o.metrics.App.Lookback <= 0 || o.metrics.App.MaxPods < 1 || o.metrics.App.MaxPVCs < 1:
|
||||
return o, errors.New("--metrics-app-lookback, --metrics-app-max-pods and --metrics-app-max-pvcs must be positive")
|
||||
case o.logs.MaxTailResume < 0:
|
||||
return o, errors.New("--logs-max-tail-resume must not be negative")
|
||||
// The proxy's deadline starts before the request reaches vmselect, so
|
||||
@@ -178,13 +189,24 @@ func run(o options) error {
|
||||
return fmt.Errorf("building client: %w", err)
|
||||
}
|
||||
|
||||
dynClient, err := dynamic.NewForConfig(restConfig)
|
||||
if err != nil {
|
||||
return fmt.Errorf("building dynamic client: %w", err)
|
||||
}
|
||||
|
||||
factory := informers.NewSharedInformerFactory(client, 0)
|
||||
namespaces := factory.Core().V1().Namespaces()
|
||||
lister := namespaces.Lister()
|
||||
appDefs := dynamicinformer.NewDynamicSharedInformerFactory(dynClient, 0).
|
||||
ForResource(observabilityproxy.ApplicationDefinitionsResource).Informer()
|
||||
if err := appDefs.AddIndexers(observabilityproxy.ApplicationPluralIndexers()); err != nil {
|
||||
return fmt.Errorf("indexing application definitions: %w", err)
|
||||
}
|
||||
factory.Start(ctx.Done())
|
||||
go appDefs.Run(ctx.Done())
|
||||
var ready atomic.Bool
|
||||
go func() {
|
||||
if cache.WaitForCacheSync(ctx.Done(), namespaces.Informer().HasSynced) {
|
||||
if cache.WaitForCacheSync(ctx.Done(), namespaces.Informer().HasSynced, appDefs.HasSynced) {
|
||||
ready.Store(true)
|
||||
}
|
||||
}()
|
||||
@@ -198,13 +220,14 @@ func run(o options) error {
|
||||
Limiter: o.limiter,
|
||||
RequestTimeout: o.requestTimeout,
|
||||
TailIdleTimeout: o.tailIdleTimeout,
|
||||
}, observabilityproxy.NewAuthorizer(client, o.authCacheTTL), observabilityproxy.NewStoreResolver(lister, namespaces.Informer().HasSynced, o.rootTenant), reg)
|
||||
}, observabilityproxy.NewAuthorizer(client, o.authCacheTTL), observabilityproxy.NewStoreResolver(lister, namespaces.Informer().HasSynced, o.rootTenant),
|
||||
observabilityproxy.NewApplicationKinds(appDefs.GetIndexer(), appDefs.HasSynced), reg)
|
||||
|
||||
probes := http.NewServeMux()
|
||||
probes.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) { _, _ = io.WriteString(w, "ok") })
|
||||
probes.HandleFunc("/readyz", func(w http.ResponseWriter, _ *http.Request) {
|
||||
if !ready.Load() {
|
||||
http.Error(w, "namespace cache not synced", http.StatusServiceUnavailable)
|
||||
http.Error(w, "namespace or application definition cache not synced", http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
_, _ = io.WriteString(w, "ok")
|
||||
|
||||
@@ -40,6 +40,9 @@ func TestParseFlagsDefaults(t *testing.T) {
|
||||
if m.Limits.MaxRange != 30*24*time.Hour || m.Limits.MaxPoints != 11000 || m.Limits.MaxTimeout != 30*time.Second {
|
||||
t.Errorf("metrics limit defaults changed: %+v", m.Limits)
|
||||
}
|
||||
if m.App.Lookback != 30*24*time.Hour || m.App.MaxPods != 500 || m.App.MaxPVCs != 500 {
|
||||
t.Errorf("application scope defaults changed: %+v", m.App)
|
||||
}
|
||||
if m.RootTenant != o.rootTenant {
|
||||
t.Errorf("metrics root tenant = %q, want --root-tenant", m.RootTenant)
|
||||
}
|
||||
@@ -63,13 +66,15 @@ func TestParseFlagsMetricsOverrides(t *testing.T) {
|
||||
o, err := parseFlags([]string{
|
||||
"--root-tenant=tenant-x", "--metrics-shortterm-store=fast", "--metrics-longterm-store=slow",
|
||||
"--metrics-shortterm-retention=24h", "--metrics-max-range=168h", "--metrics-max-points=500", "--metrics-max-timeout=10s",
|
||||
"--metrics-app-lookback=2h", "--metrics-app-max-pods=7", "--metrics-app-max-pvcs=3",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m := o.metrics
|
||||
if m.RootTenant != "tenant-x" || m.ShortTermStore != "fast" || m.LongTermStore != "slow" || m.ShortTermRetention != 24*time.Hour ||
|
||||
m.Limits.MaxRange != 168*time.Hour || m.Limits.MaxPoints != 500 || m.Limits.MaxTimeout != 10*time.Second {
|
||||
m.Limits.MaxRange != 168*time.Hour || m.Limits.MaxPoints != 500 || m.Limits.MaxTimeout != 10*time.Second ||
|
||||
m.App.Lookback != 2*time.Hour || m.App.MaxPods != 7 || m.App.MaxPVCs != 3 {
|
||||
t.Errorf("overrides not applied: %+v", m)
|
||||
}
|
||||
}
|
||||
@@ -101,6 +106,12 @@ func TestParseFlagsInvalid(t *testing.T) {
|
||||
{"--metrics-max-range=0s"},
|
||||
{"--metrics-max-points=1"},
|
||||
{"--metrics-max-timeout=0s"},
|
||||
{"--metrics-app-lookback=-1h"},
|
||||
// Every query reads some time before its window, so a zero lookback
|
||||
// would refuse every application metrics query.
|
||||
{"--metrics-app-lookback=0s"},
|
||||
{"--metrics-app-max-pods=0"},
|
||||
{"--metrics-app-max-pvcs=0"},
|
||||
{"--unknown"},
|
||||
// The proxy would cut the request while vmselect is still within
|
||||
// the timeout it was given.
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
/*
|
||||
Copyright 2026 The Cozystack Authors.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package observabilityproxy
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"slices"
|
||||
|
||||
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
|
||||
"k8s.io/apimachinery/pkg/runtime/schema"
|
||||
"k8s.io/client-go/tools/cache"
|
||||
)
|
||||
|
||||
// ApplicationDefinitionsResource is where the plural of every application
|
||||
// kind is declared (api/v1alpha1/applicationdefinitions_types.go).
|
||||
var ApplicationDefinitionsResource = schema.GroupVersionResource{Group: "cozystack.io", Version: "v1alpha1", Resource: "applicationdefinitions"}
|
||||
|
||||
// ErrKindsNotSynced means the ApplicationDefinitions cache has not been
|
||||
// filled yet, so an unknown plural cannot be told from a missing one.
|
||||
var ErrKindsNotSynced = errors.New("application definitions not synced")
|
||||
|
||||
const pluralIndex = "plural"
|
||||
|
||||
// KindLookup maps an application plural, as it appears in paths and RBAC,
|
||||
// to the kind the lineage labels carry. An unknown plural is "", nil.
|
||||
type KindLookup interface {
|
||||
Kind(plural string) (string, error)
|
||||
}
|
||||
|
||||
// ApplicationPluralIndexers indexes ApplicationDefinitions by
|
||||
// spec.application.plural; the informer feeding ApplicationKinds needs it.
|
||||
func ApplicationPluralIndexers() cache.Indexers {
|
||||
return cache.Indexers{pluralIndex: func(obj any) ([]string, error) {
|
||||
u, ok := obj.(*unstructured.Unstructured)
|
||||
if !ok {
|
||||
return nil, nil
|
||||
}
|
||||
plural, _, _ := unstructured.NestedString(u.Object, "spec", "application", "plural")
|
||||
if plural == "" {
|
||||
return nil, nil
|
||||
}
|
||||
return []string{plural}, nil
|
||||
}}
|
||||
}
|
||||
|
||||
// ApplicationKinds is the one place the proxy maps a plural to its kind;
|
||||
// both application roots use it.
|
||||
type ApplicationKinds struct {
|
||||
indexer cache.Indexer
|
||||
hasSynced func() bool
|
||||
}
|
||||
|
||||
func NewApplicationKinds(indexer cache.Indexer, hasSynced func() bool) *ApplicationKinds {
|
||||
return &ApplicationKinds{indexer: indexer, hasSynced: hasSynced}
|
||||
}
|
||||
|
||||
func (k *ApplicationKinds) Kind(plural string) (string, error) {
|
||||
if !k.hasSynced() {
|
||||
return "", ErrKindsNotSynced
|
||||
}
|
||||
objs, err := k.indexer.ByIndex(pluralIndex, plural)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
var kinds []string
|
||||
for _, obj := range objs {
|
||||
u, ok := obj.(*unstructured.Unstructured)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
kind, _, _ := unstructured.NestedString(u.Object, "spec", "application", "kind")
|
||||
if kind != "" && !slices.Contains(kinds, kind) {
|
||||
kinds = append(kinds, kind)
|
||||
}
|
||||
}
|
||||
switch len(kinds) {
|
||||
case 0:
|
||||
return "", nil
|
||||
case 1:
|
||||
return kinds[0], nil
|
||||
}
|
||||
// Picking one would scope the request to whichever definition the
|
||||
// cache happens to list first.
|
||||
slices.Sort(kinds)
|
||||
return "", fmt.Errorf("plural %q is declared for kinds %v", plural, kinds)
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
/*
|
||||
Copyright 2026 The Cozystack Authors.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package observabilityproxy
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
|
||||
"k8s.io/client-go/tools/cache"
|
||||
)
|
||||
|
||||
func appDefinition(name, kind, plural string) *unstructured.Unstructured {
|
||||
return &unstructured.Unstructured{Object: map[string]any{
|
||||
"apiVersion": "cozystack.io/v1alpha1",
|
||||
"kind": "ApplicationDefinition",
|
||||
"metadata": map[string]any{"name": name},
|
||||
"spec": map[string]any{"application": map[string]any{"kind": kind, "plural": plural, "singular": name}},
|
||||
}}
|
||||
}
|
||||
|
||||
func newTestKinds(t *testing.T, synced bool, defs ...*unstructured.Unstructured) *ApplicationKinds {
|
||||
t.Helper()
|
||||
indexer := cache.NewIndexer(cache.MetaNamespaceKeyFunc, ApplicationPluralIndexers())
|
||||
for _, d := range defs {
|
||||
if err := indexer.Add(d); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return NewApplicationKinds(indexer, func() bool { return synced })
|
||||
}
|
||||
|
||||
func TestApplicationKinds(t *testing.T) {
|
||||
kinds := newTestKinds(t, true,
|
||||
appDefinition("postgres", "Postgres", "postgreses"),
|
||||
appDefinition("vm-instance", "VMInstance", "vminstances"),
|
||||
appDefinition("broken", "", "brokens"),
|
||||
)
|
||||
for plural, want := range map[string]string{"postgreses": "Postgres", "vminstances": "VMInstance", "postgres": "", "brokens": "", "": ""} {
|
||||
got, err := kinds.Kind(plural)
|
||||
if err != nil || got != want {
|
||||
t.Errorf("Kind(%q) = %q, %v; want %q", plural, got, err, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplicationKindsNotSynced(t *testing.T) {
|
||||
kinds := newTestKinds(t, false, appDefinition("postgres", "Postgres", "postgreses"))
|
||||
if _, err := kinds.Kind("postgreses"); !errors.Is(err, ErrKindsNotSynced) {
|
||||
t.Fatalf("Kind before sync: err = %v, want ErrKindsNotSynced", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplicationKindsAmbiguous(t *testing.T) {
|
||||
kinds := newTestKinds(t, true, appDefinition("a", "Postgres", "postgreses"), appDefinition("b", "Postgresql", "postgreses"))
|
||||
if kind, err := kinds.Kind("postgreses"); err == nil {
|
||||
t.Fatalf("two kinds for one plural resolved to %q", kind)
|
||||
}
|
||||
same := newTestKinds(t, true, appDefinition("a", "Postgres", "postgreses"), appDefinition("b", "Postgres", "postgreses"))
|
||||
if kind, err := same.Kind("postgreses"); err != nil || kind != "Postgres" {
|
||||
t.Fatalf("duplicate definitions of one kind: %q, %v", kind, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,217 @@
|
||||
/*
|
||||
Copyright 2026 The Cozystack Authors.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package observabilityproxy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// AppScopeLimits bound the pod and PVC lookup of the application metrics
|
||||
// root.
|
||||
type AppScopeLimits struct {
|
||||
// Lookback is how far before the window the lookup reaches at most, so
|
||||
// pods gone before the window but inside an expression's range count.
|
||||
Lookback time.Duration
|
||||
MaxPods int
|
||||
MaxPVCs int
|
||||
}
|
||||
|
||||
// maxLookupBytes bounds a lookup answer; the caps keep a valid one far
|
||||
// below it.
|
||||
const maxLookupBytes = 8 << 20
|
||||
|
||||
// The lineage labels as the scrape class writes them on series, and as
|
||||
// kube-state-metrics writes them on kube_pod_labels and
|
||||
// kube_persistentvolumeclaim_labels.
|
||||
const (
|
||||
seriesKindLabel = "apps_cozystack_io_application_kind"
|
||||
seriesNameLabel = "apps_cozystack_io_application_name"
|
||||
ksmKindLabel = "label_" + seriesKindLabel
|
||||
ksmNameLabel = "label_" + seriesNameLabel
|
||||
)
|
||||
|
||||
func identitySelector(ns, kindLabel, nameLabel, kind, name string) string {
|
||||
return "{namespace=" + strconv.Quote(ns) + "," + kindLabel + "=" + strconv.Quote(kind) + "," + nameLabel + "=" + strconv.Quote(name) + "}"
|
||||
}
|
||||
|
||||
// nameBranch matches label against any of names within ns. With no names
|
||||
// it is "": a branch of pod=~"^()$" would match every series without a pod
|
||||
// label.
|
||||
func nameBranch(ns, label string, names []string) string {
|
||||
if len(names) == 0 {
|
||||
return ""
|
||||
}
|
||||
quoted := make([]string, len(names))
|
||||
for i, n := range names {
|
||||
quoted[i] = regexp.QuoteMeta(n)
|
||||
}
|
||||
return "{namespace=" + strconv.Quote(ns) + "," + label + "=~" + strconv.Quote("^("+strings.Join(quoted, "|")+")$") + "}"
|
||||
}
|
||||
|
||||
// applicationSelectors is the application scope: series stamped with the
|
||||
// application's identity, plus series of its pods and PVCs for those not
|
||||
// stamped (cAdvisor, kubelet volume stats). Every branch carries the
|
||||
// namespace, so their OR stays inside it.
|
||||
//
|
||||
// Within the namespace this narrows, it does not isolate: the identity and
|
||||
// the pod and PVC lists come from lineage labels that the lineage webhook
|
||||
// keeps as set on an object it finds no owner for, and the pod and PVC
|
||||
// branches match by name, so a name another application held within the
|
||||
// lookup window pulls that application's series in. Route.Reviews makes
|
||||
// the application roots require read access to the namespace root too.
|
||||
func applicationSelectors(ns, kind, name string, pods, pvcs []string) []string {
|
||||
out := []string{identitySelector(ns, seriesKindLabel, seriesNameLabel, kind, name)}
|
||||
for _, b := range []string{nameBranch(ns, "pod", pods), nameBranch(ns, "persistentvolumeclaim", pvcs)} {
|
||||
if b != "" {
|
||||
out = append(out, b)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// applicationScope looks the application's pods and PVCs up in store over
|
||||
// the samples the request reads: back by the query's own lookback, and
|
||||
// forward to the latest sample a negative offset or a subquery's
|
||||
// evaluation past its end reads. A query that reads
|
||||
// further back than the configured lookback, or whose earliest or latest
|
||||
// sample is unknown, a failed lookup, and a list over its cap all fail the
|
||||
// request, because a successful answer would silently miss the
|
||||
// application's series.
|
||||
func (s *Server) applicationScope(ctx context.Context, r *http.Request, store string, req metricsRequest, ns, kind, name string) ([]string, *HTTPError) {
|
||||
if req.last.IsZero() {
|
||||
return nil, errorf(http.StatusUnprocessableEntity, "the query reads samples at a time the application scope cannot cover (an @ anywhere in the query, a backslash or a comment outside a string, rollup_candlestick, WITH with a negative offset, an offset that is not a plain duration, or a subquery step that is not one); use the namespace root")
|
||||
}
|
||||
lim := s.cfg.Metrics.App
|
||||
if req.start.IsZero() || req.start.Before(req.from.Add(-lim.Lookback)) {
|
||||
return nil, errorf(http.StatusUnprocessableEntity, "the query reads further back than the application scope looks up pods (%s before the window, or an unknown lookback); use a shorter window or the namespace root", lim.Lookback)
|
||||
}
|
||||
start := req.start
|
||||
var lists [2][]string
|
||||
for i, l := range []struct {
|
||||
label, metric string
|
||||
max int
|
||||
}{{"pod", "kube_pod_labels", lim.MaxPods}, {"persistentvolumeclaim", "kube_persistentvolumeclaim_labels", lim.MaxPVCs}} {
|
||||
match := l.metric + identitySelector(ns, ksmKindLabel, ksmNameLabel, kind, name)
|
||||
names, err := s.lookupNames(ctx, r, store, l.label, match, ns, start, req.last, l.max)
|
||||
if err != nil {
|
||||
slog.Warn("application lookup failed", "label", l.label, "namespace", ns, "kind", kind, "name", name, "error", err)
|
||||
// The deadline can also expire while the answer is read, after
|
||||
// post has handed it over.
|
||||
if errors.Is(context.Cause(ctx), context.DeadlineExceeded) {
|
||||
return nil, errorf(http.StatusGatewayTimeout, "the application's %s lookup did not answer in time", l.label)
|
||||
}
|
||||
if herr, ok := err.(*HTTPError); ok && (herr.Code == http.StatusUnprocessableEntity || herr.Code == http.StatusGatewayTimeout) {
|
||||
return nil, herr
|
||||
}
|
||||
return nil, errorf(http.StatusBadGateway, "the application's %s lookup failed", l.label)
|
||||
}
|
||||
if len(names) > l.max {
|
||||
return nil, errorf(http.StatusUnprocessableEntity, "the application has more than %d %ss in this window; use a shorter range", l.max, l.label)
|
||||
}
|
||||
lists[i] = names
|
||||
}
|
||||
return applicationSelectors(ns, kind, name, lists[0], lists[1]), nil
|
||||
}
|
||||
|
||||
// lookupNames asks for at most limit+1 values, enough to tell a list over
|
||||
// the cap without reading all of it.
|
||||
func (s *Server) lookupNames(ctx context.Context, r *http.Request, store, label, match, ns string, start, end time.Time, limit int) ([]string, error) {
|
||||
params := url.Values{}
|
||||
params.Set("match[]", match)
|
||||
params.Set("extra_filters[]", namespaceSelector(ns))
|
||||
params.Set("start", formatTime(start))
|
||||
params.Set("end", formatTime(end))
|
||||
params.Set("limit", strconv.Itoa(limit+1))
|
||||
resp, herr := s.post(ctx, r, s.cfg.Metrics.UpstreamURL(store)+"/api/v1/label/"+label+"/values", params, "metrics store", nil)
|
||||
if herr != nil {
|
||||
return nil, herr
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
if resp.StatusCode == http.StatusUnprocessableEntity {
|
||||
return nil, lookupRefusal(label, resp.Body)
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, errorf(resp.StatusCode, "metrics store answered %s", resp.Status)
|
||||
}
|
||||
var body struct {
|
||||
Status string `json:"status"`
|
||||
IsPartial bool `json:"isPartial"`
|
||||
Data []string `json:"data"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(resp.Body, maxLookupBytes)).Decode(&body); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if body.Status != "success" {
|
||||
return nil, errorf(http.StatusBadGateway, "metrics store answered status %q", body.Status)
|
||||
}
|
||||
if len(body.Data) > limit {
|
||||
// Over the cap before dropping empty values: the store cut the
|
||||
// list at limit+1, so a shorter filtered list could hide names.
|
||||
// A partial list over the cap settles it too.
|
||||
return body.Data, nil
|
||||
}
|
||||
// A storage node that did not answer leaves names out of the list.
|
||||
if body.IsPartial {
|
||||
return nil, errorf(http.StatusBadGateway, "metrics store answered a partial %s list", label)
|
||||
}
|
||||
names := body.Data[:0]
|
||||
for _, n := range body.Data {
|
||||
if n != "" {
|
||||
names = append(names, n)
|
||||
}
|
||||
}
|
||||
return names, nil
|
||||
}
|
||||
|
||||
// seriesLimitError is the message vmstorage gives when a search matches
|
||||
// more series than the -search.max* flags allow (lib/storage/index_db.go);
|
||||
// cluster vmselect wraps it in its own context, hence a substring match.
|
||||
const seriesLimitError = "the number of matching timeseries exceeds"
|
||||
|
||||
// lookupRefusal reads a vmselect 422. vmselect answers 422 to any error
|
||||
// without a status of its own (httpserver.SendPrometheusError), storage
|
||||
// failures and its own timeout included, so only the series limit, which
|
||||
// a narrower window fixes, stays a 422 for the client.
|
||||
func lookupRefusal(label string, body io.Reader) *HTTPError {
|
||||
var answer struct {
|
||||
Error string `json:"error"`
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(body, 64<<10)).Decode(&answer)
|
||||
switch {
|
||||
case strings.Contains(answer.Error, seriesLimitError):
|
||||
// From the limit on: what precedes it is cluster vmselect's context,
|
||||
// which can name vmstorage nodes.
|
||||
limit := answer.Error[strings.Index(answer.Error, seriesLimitError):]
|
||||
return errorf(http.StatusUnprocessableEntity, "the application's %s lookup: %s", label, limit)
|
||||
case strings.Contains(answer.Error, "timeout exceeded"):
|
||||
return errorf(http.StatusGatewayTimeout, "the application's %s lookup did not answer in time", label)
|
||||
default:
|
||||
// The text may name vmstorage nodes, so it stays in the log.
|
||||
slog.Warn("application lookup refused by the store", "label", label, "error", answer.Error)
|
||||
return errorf(http.StatusBadGateway, "the application's %s lookup failed", label)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
/*
|
||||
Copyright 2026 The Cozystack Authors.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package observabilityproxy
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
const wantIdentityBranch = `{namespace="tenant-a",apps_cozystack_io_application_kind="Postgres",apps_cozystack_io_application_name="db.main"}`
|
||||
|
||||
func TestApplicationSelectors(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
pods, pvcs []string
|
||||
want []string
|
||||
}{
|
||||
{"identity only", nil, nil, []string{wantIdentityBranch}},
|
||||
{"pods", []string{"db-main-1", "db.main-2"}, nil, []string{
|
||||
wantIdentityBranch,
|
||||
`{namespace="tenant-a",pod=~"^(db-main-1|db\\.main-2)$"}`,
|
||||
}},
|
||||
{"pvcs", nil, []string{"data-db-main-1"}, []string{
|
||||
wantIdentityBranch,
|
||||
`{namespace="tenant-a",persistentvolumeclaim=~"^(data-db-main-1)$"}`,
|
||||
}},
|
||||
{"all three", []string{"p"}, []string{"v"}, []string{
|
||||
wantIdentityBranch,
|
||||
`{namespace="tenant-a",pod=~"^(p)$"}`,
|
||||
`{namespace="tenant-a",persistentvolumeclaim=~"^(v)$"}`,
|
||||
}},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if got := applicationSelectors("tenant-a", "Postgres", "db.main", tt.pods, tt.pvcs); !slices.Equal(got, tt.want) {
|
||||
t.Fatalf("got %q\nwant %q", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// DNS-1123 keeps most metacharacters out of names, but the names come
|
||||
// back from the store, so the encoder must not rely on that.
|
||||
func TestNameBranchEscapes(t *testing.T) {
|
||||
names := []string{`a.b`, `x|.*`, `)|(.*`, `q"} or {namespace=~".+`, `back\slash`, `$^+?[]{}`}
|
||||
sel := nameBranch("tenant-a", "pod", names)
|
||||
m := regexp.MustCompile(`^\{namespace="tenant-a",pod=~(".*")\}$`).FindStringSubmatch(sel)
|
||||
if m == nil {
|
||||
t.Fatalf("selector %s is not one namespace and one pod matcher", sel)
|
||||
}
|
||||
pattern, err := strconv.Unquote(m[1])
|
||||
if err != nil {
|
||||
t.Fatalf("pod matcher %s does not unquote: %v", m[1], err)
|
||||
}
|
||||
re := regexp.MustCompile(pattern)
|
||||
for _, n := range names {
|
||||
if !re.MatchString(n) {
|
||||
t.Errorf("%q does not match its own branch", n)
|
||||
}
|
||||
}
|
||||
for _, other := range []string{"axb", "x", "anything", "tenant-b", "", `back\\slash`} {
|
||||
if re.MatchString(other) {
|
||||
t.Errorf("branch matches %q", other)
|
||||
}
|
||||
}
|
||||
if nameBranch("tenant-a", "pod", nil) != "" {
|
||||
t.Error("an empty list produced a branch, which would match series without the label")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetricsRequestScope(t *testing.T) {
|
||||
branches := []string{"{a}", "{b}"}
|
||||
labels := buildMetrics(t, "api/v1/labels", url.Values{})
|
||||
labels.setScope(branches)
|
||||
if got := labels.params["extra_filters[]"]; !slices.Equal(got, branches) {
|
||||
t.Errorf("extra_filters[] = %q", got)
|
||||
}
|
||||
if got := labels.params["match[]"]; !slices.Equal(got, branches) {
|
||||
t.Errorf("default match[] = %q, want the branches", got)
|
||||
}
|
||||
client := buildMetrics(t, "api/v1/labels", url.Values{"match[]": {"up"}})
|
||||
client.setScope(branches)
|
||||
if got := client.params["match[]"]; !slices.Equal(got, []string{"up"}) {
|
||||
t.Errorf("client match[] replaced: %q", got)
|
||||
}
|
||||
query := buildMetrics(t, "api/v1/query", url.Values{"query": {"up"}})
|
||||
query.setScope(branches)
|
||||
if query.params.Has("match[]") || !slices.Equal(query.params["extra_filters[]"], branches) {
|
||||
t.Errorf("query params = %v", query.params)
|
||||
}
|
||||
if !query.from.Equal(testNow) || !query.to.Equal(testNow) {
|
||||
t.Errorf("instant window = [%s, %s], want now", query.from, query.to)
|
||||
}
|
||||
rng := buildMetrics(t, "api/v1/query_range", url.Values{"query": {"up"}, "start": {unix(testNow.Add(-time.Hour))}, "end": {unix(testNow)}, "step": {"60"}})
|
||||
if !rng.from.Equal(testNow.Add(-time.Hour)) || !rng.to.Equal(testNow) {
|
||||
t.Errorf("range window = [%s, %s]", rng.from, rng.to)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLookupRefusalKeepsStoreErrorsInternal: a store error other than the
|
||||
// series limit may name vmstorage nodes; it goes to the log, not into a
|
||||
// message that could reach the client.
|
||||
func TestLookupRefusalKeepsStoreErrorsInternal(t *testing.T) {
|
||||
body := strings.NewReader(`{"status":"error","error":"cannot fetch from vmstorage-longterm-0.vmstorage:8401: connection refused"}`)
|
||||
herr := lookupRefusal("pod", body)
|
||||
if herr.Code != http.StatusBadGateway || strings.Contains(herr.Message, "vmstorage") {
|
||||
t.Fatalf("lookupRefusal = %d %q, want 502 without the store's text", herr.Code, herr.Message)
|
||||
}
|
||||
}
|
||||
@@ -41,14 +41,20 @@ const latencyOffset = 30 * time.Second
|
||||
|
||||
// queryDurationPart is the subset of the MetricsQL duration grammar
|
||||
// (scanSingleDuration in metricsql) the scan understands. Step multiples
|
||||
// (5i, $__interval), negative parts and upper-case units are left out on
|
||||
// purpose: anything else makes the bound unknown.
|
||||
// (5i, $__interval) and negative parts are left out on purpose: anything
|
||||
// else makes the bound unknown. Offsets are scanned lower-cased, as
|
||||
// metricsql reads units in any case; windows and steps in brackets are
|
||||
// not, so an upper-case unit there is unknown.
|
||||
var (
|
||||
queryDurationPart = regexp.MustCompile(`(\d+(?:\.\d+)?)(ms|s|m|h|d|w|y)`)
|
||||
queryDuration = regexp.MustCompile(`^(?:\d+(?:\.\d+)?(?:ms|s|m|h|d|w|y))+$`)
|
||||
querySeconds = regexp.MustCompile(`^\d+(?:\.\d+)?$`)
|
||||
)
|
||||
|
||||
// offsetTerminators may follow an offset's duration: whitespace, a
|
||||
// closing bracket or separator, or a binary operator other than minus.
|
||||
const offsetTerminators = " \t\n\r)]},+*/^%<>=!"
|
||||
|
||||
var queryDurationUnits = map[string]time.Duration{
|
||||
"ms": time.Millisecond, "s": time.Second, "m": time.Minute, "h": time.Hour,
|
||||
"d": 24 * time.Hour, "w": 7 * 24 * time.Hour, "y": 365 * 24 * time.Hour,
|
||||
@@ -113,10 +119,88 @@ func queryLookback(query string, step time.Duration) (time.Duration, bool) {
|
||||
sum += d
|
||||
}
|
||||
lower := strings.ToLower(query)
|
||||
back, _, ok := scanOffsets(lower)
|
||||
if !ok {
|
||||
return 0, false
|
||||
}
|
||||
sum += back
|
||||
if hasKeyword(lower, "with") {
|
||||
return 0, false
|
||||
}
|
||||
n := float64(subqueries + strings.Count(query, "("))
|
||||
sum += (2+3*n)*maxStep + (1+n)*float64(silenceInterval)
|
||||
if sum > float64(maxMetricsDuration) {
|
||||
return 0, false
|
||||
}
|
||||
return time.Duration(sum), true
|
||||
}
|
||||
|
||||
// queryLookahead is an upper bound on how far after its evaluation window
|
||||
// a query reads, from the same lexical scan, with step the query's step.
|
||||
// A negative offset reads forward; the negative offsets are summed, which
|
||||
// covers nesting. vmselect also evaluates a subquery up to its end plus
|
||||
// its step, rounded up to the step (evalRollupFuncWithSubquery), and
|
||||
// range_* transforms and keep_next_value carry those later values back
|
||||
// onto earlier points, so each subquery level adds two steps; as for the
|
||||
// lookback, every "(" counts as a possible implicit subquery and each
|
||||
// term uses the largest step seen. Any @ makes the bound unknown, as it
|
||||
// does for queryLookback, and so does WITH, which can apply one offset
|
||||
// several times, and a backslash or a comment outside a string, which can
|
||||
// hide what the scan looks for. vmselect also evaluates
|
||||
// rollup_candlestick one step ahead of the evaluation time
|
||||
// (evalRollupFuncWithoutAt), so it makes the bound unknown too.
|
||||
func queryLookahead(query string, step time.Duration) (time.Duration, bool) {
|
||||
if strings.Contains(query, "@") || unscannableOutsideString(query) {
|
||||
return 0, false
|
||||
}
|
||||
lower := strings.ToLower(query)
|
||||
if hasKeyword(lower, "rollup_candlestick") {
|
||||
return 0, false
|
||||
}
|
||||
_, ahead, ok := scanOffsets(lower)
|
||||
if !ok || ahead > 0 && hasKeyword(lower, "with") {
|
||||
return 0, false
|
||||
}
|
||||
subqueries := 0
|
||||
maxStep := float64(step)
|
||||
for rest := query; ; {
|
||||
open := strings.IndexByte(rest, '[')
|
||||
if open < 0 {
|
||||
break
|
||||
}
|
||||
closing := strings.IndexByte(rest[open:], ']')
|
||||
if closing < 0 {
|
||||
return 0, false
|
||||
}
|
||||
_, subStep, isSubquery := strings.Cut(rest[open+1:open+closing], ":")
|
||||
rest = rest[open+closing+1:]
|
||||
if !isSubquery {
|
||||
continue
|
||||
}
|
||||
subqueries++
|
||||
if strings.TrimSpace(subStep) != "" {
|
||||
d, ok := parseQueryDuration(subStep)
|
||||
if !ok {
|
||||
return 0, false
|
||||
}
|
||||
maxStep = max(maxStep, d)
|
||||
}
|
||||
}
|
||||
ahead += 2 * float64(subqueries+strings.Count(query, "(")) * maxStep
|
||||
if ahead > float64(maxMetricsDuration) {
|
||||
return 0, false
|
||||
}
|
||||
return time.Duration(ahead), true
|
||||
}
|
||||
|
||||
// scanOffsets sums every offset of the lowercased query, the positive ones
|
||||
// into back and the negative ones into ahead, in nanoseconds; ok is false
|
||||
// when one is not a plain duration.
|
||||
func scanOffsets(lower string) (back, ahead float64, ok bool) {
|
||||
for i := 0; ; {
|
||||
k := strings.Index(lower[i:], "offset")
|
||||
if k < 0 {
|
||||
break
|
||||
return back, ahead, true
|
||||
}
|
||||
k += i
|
||||
i = k + len("offset")
|
||||
@@ -132,23 +216,53 @@ func queryLookback(query string, step time.Duration) (time.Duration, bool) {
|
||||
if end < 0 {
|
||||
end = len(arg)
|
||||
}
|
||||
// metricsql lexes 1h-30m as one duration whose parts after the
|
||||
// first negative one all subtract (DurationValue), so -1h-30m is
|
||||
// -90m and 1h-2h is -1h; the scan does not read such compounds.
|
||||
// metricsql continues a duration token in more forms than the scan
|
||||
// reads: after a minus (-30m, -1.h, -.5h) and with a trailing
|
||||
// $__interval, all summed by DurationValue. Only what certainly
|
||||
// ends the token is accepted after the duration.
|
||||
if end < len(arg) && !strings.ContainsRune(offsetTerminators, rune(arg[end])) {
|
||||
return 0, 0, false
|
||||
}
|
||||
d, ok := parseQueryDuration(arg[:end])
|
||||
if !ok {
|
||||
return 0, false
|
||||
return 0, 0, false
|
||||
}
|
||||
if !negative {
|
||||
sum += d
|
||||
if negative {
|
||||
ahead += d
|
||||
} else {
|
||||
back += d
|
||||
}
|
||||
}
|
||||
if hasKeyword(lower, "with") {
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// unscannableOutsideString reports a backslash or a "#" outside a string
|
||||
// literal. metricsql unescapes identifiers (unescapeIdent), so
|
||||
// rollup\_candlestick or offs\et name what the lexical scan looks for
|
||||
// without its spelling; "#" starts a comment to the end of the line, and
|
||||
// quotes or brackets inside one would mislead the string tracking here and
|
||||
// the bracket and offset scans. Inside any of the three quotes a backslash
|
||||
// skips the next character, as metricsql's scanString does: an escaped
|
||||
// closing quote does not end the string, backquotes included (a quoted
|
||||
// label name keeps such an escape).
|
||||
func unscannableOutsideString(query string) bool {
|
||||
var quote byte
|
||||
for i := 0; i < len(query); i++ {
|
||||
c := query[i]
|
||||
switch {
|
||||
case quote == 0 && (c == '\\' || c == '#'):
|
||||
return true
|
||||
case quote == 0 && (c == '"' || c == '\'' || c == '`'):
|
||||
quote = c
|
||||
case quote != 0 && c == '\\':
|
||||
i++
|
||||
case quote != 0 && c == quote:
|
||||
quote = 0
|
||||
}
|
||||
}
|
||||
n := float64(subqueries + strings.Count(query, "("))
|
||||
sum += (2+3*n)*maxStep + (1+n)*float64(silenceInterval)
|
||||
if sum > float64(maxMetricsDuration) {
|
||||
return 0, false
|
||||
}
|
||||
return time.Duration(sum), true
|
||||
return false
|
||||
}
|
||||
|
||||
// isKeyword reports whether lower[start:end] stands on its own rather than
|
||||
|
||||
@@ -46,6 +46,7 @@ type MetricsConfig struct {
|
||||
LongTermStore string
|
||||
ShortTermRetention time.Duration
|
||||
Limits MetricsLimits
|
||||
App AppScopeLimits
|
||||
// UpstreamURL maps a store name to its Prometheus API base URL, without
|
||||
// a trailing slash; nil means the in-cluster vmselect Service.
|
||||
UpstreamURL func(store string) string
|
||||
@@ -77,6 +78,23 @@ type metricsRequest struct {
|
||||
// start is the earliest instant the request reads, for the store choice;
|
||||
// zero when that is unknown, which picks the long-term store.
|
||||
start time.Time
|
||||
// from and to are the window the query is evaluated over, without the
|
||||
// lookback of its expression.
|
||||
from, to time.Time
|
||||
// last is the latest instant the request reads, past to by a negative
|
||||
// offset; zero when that is unknown.
|
||||
last time.Time
|
||||
// scopedMatch is set when match[] holds the scope, not client matchers.
|
||||
scopedMatch bool
|
||||
}
|
||||
|
||||
// setScope replaces the namespace scope with the given selectors, which
|
||||
// vmselect ORs.
|
||||
func (req *metricsRequest) setScope(selectors []string) {
|
||||
req.params["extra_filters[]"] = selectors
|
||||
if req.scopedMatch {
|
||||
req.params["match[]"] = selectors
|
||||
}
|
||||
}
|
||||
|
||||
// namespaceSelector quotes ns the way MetricsQL unquotes a label value
|
||||
@@ -116,8 +134,11 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti
|
||||
return req, herr
|
||||
}
|
||||
at = t
|
||||
out.Set("time", formatTime(t))
|
||||
}
|
||||
// The default is forwarded too: the application lookup is sized
|
||||
// for at, and vmselect's own clock reads later, by as long as the
|
||||
// lookups took.
|
||||
out.Set("time", formatTime(at))
|
||||
step := instantDefaultStep
|
||||
if v := in.Get("step"); v != "" {
|
||||
var herr *HTTPError
|
||||
@@ -126,7 +147,15 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti
|
||||
}
|
||||
out.Set("step", formatSeconds(step))
|
||||
}
|
||||
req.start = queryStart(out.Get("query"), at.Add(-latencyOffset), step)
|
||||
// vmselect evaluates an instant query whose time is within the
|
||||
// latency offset of its own clock at that clock minus the offset
|
||||
// (queryOffset in prometheus.go), so the evaluation instant lies in
|
||||
// (time-2*offset, time]; the earlier end needs no clock to agree.
|
||||
req.start = queryStart(out.Get("query"), at.Add(-2*latencyOffset), step)
|
||||
req.from, req.to = at, at
|
||||
if ahead, ok := queryLookahead(out.Get("query"), step); ok {
|
||||
req.last = at.Add(ahead)
|
||||
}
|
||||
return req, nil
|
||||
|
||||
case "api/v1/query_range":
|
||||
@@ -150,6 +179,11 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti
|
||||
return req, errorf(http.StatusUnprocessableEntity, "%d points per series exceed the limit of %d; raise step", points, lim.MaxPoints)
|
||||
}
|
||||
req.start = queryStart(out.Get("query"), start, step)
|
||||
req.from, req.to = start, end
|
||||
// vmselect rounds end up to a whole step (AdjustStartEnd).
|
||||
if ahead, ok := queryLookahead(out.Get("query"), step); ok {
|
||||
req.last = end.Add(step + ahead)
|
||||
}
|
||||
out.Set("start", formatTime(start))
|
||||
out.Set("end", formatTime(end))
|
||||
out.Set("step", formatSeconds(step))
|
||||
@@ -172,7 +206,7 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti
|
||||
if herr != nil {
|
||||
return req, herr
|
||||
}
|
||||
req.start = start
|
||||
req.start, req.from, req.to, req.last = start, start, end, end
|
||||
out.Set("start", formatTime(start))
|
||||
out.Set("end", formatTime(end))
|
||||
|
||||
@@ -194,6 +228,7 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti
|
||||
// extra_filters[] on the label APIs unless a match[] is present
|
||||
// (getCommonParamsInternal), so the scope is repeated here.
|
||||
out.Set("match[]", namespaceSelector(namespace))
|
||||
req.scopedMatch = true
|
||||
}
|
||||
return req, nil
|
||||
}
|
||||
|
||||
@@ -227,7 +227,7 @@ func TestBuildMetricsParamsCanonicalForms(t *testing.T) {
|
||||
if req.params.Get("end") != "2026-11-04T12:00:00.000Z" || req.params.Get("start") != "2026-11-04T11:55:00.000Z" {
|
||||
t.Errorf("labels window = %s..%s, want the last 5m", req.params.Get("start"), req.params.Get("end"))
|
||||
}
|
||||
if q := buildMetrics(t, "api/v1/query", url.Values{"query": {"up"}}); q.params.Has("time") || !q.start.Equal(testNow.Add(-15*time.Minute-30*time.Second)) {
|
||||
if q := buildMetrics(t, "api/v1/query", url.Values{"query": {"up"}}); q.params.Get("time") != "2026-11-04T12:00:00.000Z" || !q.start.Equal(testNow.Add(-15*time.Minute-2*latencyOffset)) {
|
||||
t.Errorf("instant query without time: params %v, start %v", q.params, q.start)
|
||||
}
|
||||
}
|
||||
@@ -429,3 +429,138 @@ func TestBuildMetricsParamsPointsAsVmselectCounts(t *testing.T) {
|
||||
t.Fatalf("11001 points answered %d, want 422", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueryLookahead(t *testing.T) {
|
||||
tests := []struct {
|
||||
query string
|
||||
want time.Duration
|
||||
}{
|
||||
{`up`, 0},
|
||||
{`x offset 1h`, 0},
|
||||
{`x offset -1h`, time.Hour},
|
||||
{`x OFFSET - 90m`, 90 * time.Minute},
|
||||
// Each "(" may be an implicit subquery, which vmselect evaluates
|
||||
// up to two steps past its end; here the step is 1m.
|
||||
{`rate(x[5m] offset -30m) offset -1h`, 92 * time.Minute},
|
||||
{`x offset -1h + y offset 2h`, time.Hour},
|
||||
{`node_offset_seconds - 1`, 0},
|
||||
// An escape inside a string literal is only part of a label value.
|
||||
{`x{a=~"a\\.b"}`, 0},
|
||||
{`x{a="\"rollup\\_candlestick\""}`, 0},
|
||||
// metricsql lexes a duration as one token, so a sign with spaces,
|
||||
// or a plus, is an operator after the offset, not part of it.
|
||||
{`x offset 1h+30m`, 0},
|
||||
{`x offset 1h - 30m`, 0},
|
||||
{`x offset -1h + 30m`, time.Hour},
|
||||
{`x offset -1h30m`, 90 * time.Minute},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.query, func(t *testing.T) {
|
||||
got, ok := queryLookahead(tt.query, time.Minute)
|
||||
if !ok || got != tt.want {
|
||||
t.Fatalf("queryLookahead = %s, %v; want %s", got, ok, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
for _, q := range []string{
|
||||
`x @ 1700000000`,
|
||||
// The lookback scan cannot bound any @, so the application root
|
||||
// refuses every one; the lookahead does not tell them apart.
|
||||
`x @ end()`,
|
||||
`x @ start() offset -1h`,
|
||||
`x @ (end() - 1h)`,
|
||||
`x{a="a@b"}`,
|
||||
`x offset -5i`,
|
||||
// Compound durations with a negative part (metricsql DurationValue):
|
||||
// -1h-30m is -90m and 1h-2h is -1h, neither of which the scan reads.
|
||||
`x offset -1h-30m`,
|
||||
`x offset - 1h-30m`,
|
||||
`x offset 1h-2h`,
|
||||
`x offset 1h-30m`,
|
||||
// metricsql continues a duration with -1.h or -.5h too, so any minus
|
||||
// right after the duration is refused rather than told apart.
|
||||
`x offset 5m-1.h`,
|
||||
`x offset 5m-.5h`,
|
||||
`x offset 1h-1`,
|
||||
`x offset 5m-y`,
|
||||
// metricsql also reads a trailing $__interval as part of the
|
||||
// duration: -1h$__interval is -(1h + step).
|
||||
`x offset -1h$__interval`,
|
||||
`rate(x[5m]) offset -1h$__interval`,
|
||||
// A subquery step the scan cannot read, an unclosed bracket, and a
|
||||
// sum past the duration range.
|
||||
`rate(x[1h:5i])`,
|
||||
`rate(x[1h:$__interval])`,
|
||||
`rate(x[1h`,
|
||||
`max_over_time(x[1h:100y])`,
|
||||
`rate(x[5m] offset 1h-30m)`,
|
||||
// metricsql unescapes identifiers (unescapeIdent), so an escaped
|
||||
// rollup_candlestick, or any escape outside a string, could hide
|
||||
// what the scan looks for.
|
||||
`rollup\_candlestick(x + y)`,
|
||||
`rollup_candle\stick(x)`,
|
||||
`rollup\x5fcandlestick(x)`,
|
||||
`x offs\et -1h`,
|
||||
// A comment runs to the end of the line, so quotes in it would
|
||||
// confuse the string tracking and brackets in it the scans.
|
||||
"x # '\n+ rollup\\_candlestick(y) # '",
|
||||
"x # \"\n+ rollup\\_candlestick(y) # \"",
|
||||
"rate(x[5m]) # comment",
|
||||
// metricsql lets a backslash escape the closing quote in backquotes
|
||||
// too (scanString), and quoted label names keep such escapes, so
|
||||
// a backquoted string must not end at an escaped backquote.
|
||||
"container_cpu_usage_seconds_total{`\\``=\"\"} or rollup\\_candlestick(container_cpu_usage_seconds_total offset -5m+1h)",
|
||||
"{`\\`a`} or rollup\\_candlestick(container_cpu_usage_seconds_total offset -5m+1h)",
|
||||
// vmselect evaluates rollup_candlestick one step ahead
|
||||
// (evalRollupFuncWithoutAt), whatever the offset.
|
||||
`rollup_candlestick(x)`,
|
||||
`ROLLUP_CANDLESTICK(x[5m])`,
|
||||
`sum(rollup_candlestick(x[1h:5m]))`,
|
||||
`x offset -$__interval`,
|
||||
`with (f(a) = a offset -1h) f(f(x))`,
|
||||
} {
|
||||
if got, ok := queryLookahead(q, time.Minute); ok {
|
||||
t.Errorf("queryLookahead(%q) = %s, want unknown", q, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestQueryLookaheadSubquery: vmselect evaluates a subquery up to its end
|
||||
// plus one subquery step, rounded up to the step (evalRollupFuncWithSubquery),
|
||||
// and range_last and the other range_* transforms copy the last of those
|
||||
// values back onto earlier points, so a series that starts after the
|
||||
// evaluation time can reach the answer.
|
||||
func TestQueryLookaheadSubquery(t *testing.T) {
|
||||
// One explicit subquery and one "(": two levels of two steps, at the
|
||||
// subquery's step, or at the query's step when the subquery has none.
|
||||
for q, want := range map[string]time.Duration{
|
||||
`max_over_time(x[1h:1h])`: 4 * time.Hour,
|
||||
`max_over_time(x[1h:])`: 4 * time.Minute,
|
||||
} {
|
||||
if got, ok := queryLookahead(q, time.Minute); !ok || got != want {
|
||||
t.Errorf("queryLookahead(%q) = %s, %v; want %s", q, got, ok, want)
|
||||
}
|
||||
}
|
||||
for _, q := range []string{`max_over_time(range_last(x)[1h:1h])`, `max_over_time(keep_next_value(x)[1h:1h])`} {
|
||||
got, ok := queryLookahead(q, time.Minute)
|
||||
if !ok || got < 2*time.Hour {
|
||||
t.Errorf("queryLookahead(%q) = %s, %v; want at least two subquery steps", q, got, ok)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestBuildMetricsParamsInstantStartCoversLatencyOffset: vmselect evaluates
|
||||
// an instant query whose time is within 30s of its clock at that clock
|
||||
// minus 30s, anywhere in (time-60s, time], so the earliest read the store
|
||||
// choice and the lookup rely on starts from time-60s.
|
||||
func TestBuildMetricsParamsInstantStartCoversLatencyOffset(t *testing.T) {
|
||||
at := testNow.Add(29 * time.Second)
|
||||
req := buildMetrics(t, "api/v1/query", url.Values{"query": {"x"}, "time": {at.Format(time.RFC3339)}, "step": {"1s"}})
|
||||
lookback, ok := queryLookback("x", time.Second)
|
||||
if !ok {
|
||||
t.Fatal("lookback of x unknown")
|
||||
}
|
||||
if want := at.Add(-2*latencyOffset - lookback); req.start.After(want) {
|
||||
t.Fatalf("start = %s, want at most %s", req.start, want)
|
||||
}
|
||||
}
|
||||
@@ -131,8 +131,28 @@ func ParseRoute(escapedPath string) (Route, *HTTPError) {
|
||||
return r, nil
|
||||
}
|
||||
|
||||
// ResourceAttributes is the single SubjectAccessReview a route requires. It
|
||||
// depends on the path only, so a POST query authorizes exactly like a GET.
|
||||
// Reviews lists every access a request on the route needs. An application
|
||||
// root also needs read access to its namespace root, because its scope
|
||||
// narrows within the namespace but does not isolate: on both roots the
|
||||
// identity comes from lineage labels that the lineage webhook keeps as set
|
||||
// on an object it finds no owner for, so anyone who can create a pod there
|
||||
// can label it as the application; on the metrics root the pod and PVC
|
||||
// branches also match by name across the window, so a name another
|
||||
// application held earlier reaches that application's series. A grant
|
||||
// narrowed by resourceNames would read more than it names.
|
||||
func (r Route) Reviews() []authorizationv1.ResourceAttributes {
|
||||
switch r.Kind {
|
||||
case ApplicationLog:
|
||||
return []authorizationv1.ResourceAttributes{Route{Kind: NamespaceLogs, Namespace: r.Namespace}.ResourceAttributes(), r.ResourceAttributes()}
|
||||
case ApplicationMetrics:
|
||||
return []authorizationv1.ResourceAttributes{Route{Kind: NamespaceMetrics, Namespace: r.Namespace}.ResourceAttributes(), r.ResourceAttributes()}
|
||||
}
|
||||
return []authorizationv1.ResourceAttributes{r.ResourceAttributes()}
|
||||
}
|
||||
|
||||
// ResourceAttributes is the SubjectAccessReview of the route's own
|
||||
// resource. It depends on the path only, so a POST query authorizes
|
||||
// exactly like a GET.
|
||||
func (r Route) ResourceAttributes() authorizationv1.ResourceAttributes {
|
||||
switch r.Kind {
|
||||
case NamespaceLogs:
|
||||
|
||||
@@ -18,6 +18,7 @@ package observabilityproxy
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"slices"
|
||||
"testing"
|
||||
|
||||
authorizationv1 "k8s.io/api/authorization/v1"
|
||||
@@ -130,3 +131,26 @@ func TestRouteResourceAttributes(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRouteReviews(t *testing.T) {
|
||||
nsLogs := authorizationv1.ResourceAttributes{Verb: "list", Group: "cozystack.io", Resource: "logs", Namespace: "a"}
|
||||
nsMetrics := authorizationv1.ResourceAttributes{Verb: "list", Group: "cozystack.io", Resource: "metrics", Namespace: "a"}
|
||||
tests := []struct {
|
||||
route Route
|
||||
want []authorizationv1.ResourceAttributes
|
||||
}{
|
||||
{Route{Kind: NamespaceLogs, Namespace: "a"}, []authorizationv1.ResourceAttributes{nsLogs}},
|
||||
{Route{Kind: NamespaceMetrics, Namespace: "a"}, []authorizationv1.ResourceAttributes{nsMetrics}},
|
||||
{Route{Kind: ApplicationLog, Namespace: "a", Plural: "postgreses", Name: "db"}, []authorizationv1.ResourceAttributes{
|
||||
nsLogs, {Verb: "get", Group: "apps.cozystack.io", Resource: "postgreses", Subresource: "log", Name: "db", Namespace: "a"},
|
||||
}},
|
||||
{Route{Kind: ApplicationMetrics, Namespace: "a", Plural: "postgreses", Name: "db"}, []authorizationv1.ResourceAttributes{
|
||||
nsMetrics, {Verb: "get", Group: "apps.cozystack.io", Resource: "postgreses", Subresource: "metrics", Name: "db", Namespace: "a"},
|
||||
}},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
if got := tt.route.Reviews(); !slices.Equal(got, tt.want) {
|
||||
t.Errorf("%v.Reviews() = %+v, want %+v", tt.route.Kind, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -73,13 +73,14 @@ type Server struct {
|
||||
cfg Config
|
||||
auth *Authorizer
|
||||
stores StoreLookup
|
||||
kinds KindLookup
|
||||
client *http.Client
|
||||
limits *limiter
|
||||
now func() time.Time
|
||||
requests *prometheus.CounterVec
|
||||
}
|
||||
|
||||
func NewServer(cfg Config, auth *Authorizer, stores StoreLookup, reg prometheus.Registerer) *Server {
|
||||
func NewServer(cfg Config, auth *Authorizer, stores StoreLookup, kinds KindLookup, reg prometheus.Registerer) *Server {
|
||||
if cfg.UpstreamURL == nil {
|
||||
domain := cfg.ClusterDomain
|
||||
cfg.UpstreamURL = func(store string) string {
|
||||
@@ -96,6 +97,7 @@ func NewServer(cfg Config, auth *Authorizer, stores StoreLookup, reg prometheus.
|
||||
cfg: cfg,
|
||||
auth: auth,
|
||||
stores: stores,
|
||||
kinds: kinds,
|
||||
// No client timeout: it would cut live tails. Non-tail requests are
|
||||
// bounded by a context deadline instead.
|
||||
client: &http.Client{Transport: http.DefaultTransport.(*http.Transport).Clone()},
|
||||
@@ -172,12 +174,7 @@ func writeError(w http.ResponseWriter, e *HTTPError) {
|
||||
http.Error(w, e.Message, e.Code)
|
||||
}
|
||||
|
||||
// serve answers a root it has no handler for before authenticating, so an
|
||||
// unknown path costs no review calls.
|
||||
func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTTPError {
|
||||
if route.Kind != NamespaceLogs && route.Kind != NamespaceMetrics {
|
||||
return errorf(http.StatusNotFound, "not found")
|
||||
}
|
||||
user, herr := s.authorize(r, route)
|
||||
if herr != nil {
|
||||
return herr
|
||||
@@ -190,8 +187,14 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT
|
||||
w.Header().Set("Retry-After", strconv.FormatInt(int64(math.Ceil(wait.Seconds())), 10))
|
||||
return errorf(http.StatusTooManyRequests, "request rate limit exceeded")
|
||||
}
|
||||
if route.Kind == NamespaceMetrics {
|
||||
return s.serveMetrics(w, r, route, user)
|
||||
var kind string
|
||||
if route.Kind == ApplicationLog || route.Kind == ApplicationMetrics {
|
||||
if kind, herr = s.applicationKind(route.Plural); herr != nil {
|
||||
return herr
|
||||
}
|
||||
}
|
||||
if route.Kind == NamespaceMetrics || route.Kind == ApplicationMetrics {
|
||||
return s.serveMetrics(w, r, route, user, kind)
|
||||
}
|
||||
endpoint, ok := strings.CutPrefix(route.Endpoint, "select/logsql/")
|
||||
if !ok {
|
||||
@@ -214,6 +217,9 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT
|
||||
if herr != nil {
|
||||
return herr
|
||||
}
|
||||
if kind != "" {
|
||||
params.Add("extra_stream_filters", applicationStreamFilter(kind, route.Name))
|
||||
}
|
||||
|
||||
store, err := s.stores.Resolve(route.Namespace)
|
||||
if err != nil {
|
||||
@@ -279,7 +285,9 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Server) serveMetrics(w http.ResponseWriter, r *http.Request, route Route, user string) *HTTPError {
|
||||
// serveMetrics scopes to the namespace, or with kind set to the application
|
||||
// route.Name of that kind within it.
|
||||
func (s *Server) serveMetrics(w http.ResponseWriter, r *http.Request, route Route, user, kind string) *HTTPError {
|
||||
release, ok := s.limits.acquire(user)
|
||||
if !ok {
|
||||
return tooManyConcurrent(w)
|
||||
@@ -301,6 +309,13 @@ func (s *Server) serveMetrics(w http.ResponseWriter, r *http.Request, route Rout
|
||||
defer cancel()
|
||||
}
|
||||
store := s.cfg.Metrics.store(req.start, now)
|
||||
if kind != "" {
|
||||
selectors, herr := s.applicationScope(ctx, r, store, req, route.Namespace, kind, route.Name)
|
||||
if herr != nil {
|
||||
return herr
|
||||
}
|
||||
req.setScope(selectors)
|
||||
}
|
||||
resp, herr := s.post(ctx, r, s.cfg.Metrics.UpstreamURL(store)+"/"+req.path, req.params, "metrics store", nil)
|
||||
if herr != nil {
|
||||
return herr
|
||||
@@ -322,6 +337,26 @@ func (s *Server) serveMetrics(w http.ResponseWriter, r *http.Request, route Rout
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Server) applicationKind(plural string) (string, *HTTPError) {
|
||||
kind, err := s.kinds.Kind(plural)
|
||||
if err != nil {
|
||||
slog.Error("mapping application plural to kind", "plural", plural, "error", err)
|
||||
return "", errorf(http.StatusServiceUnavailable, "cannot resolve the application kind")
|
||||
}
|
||||
if kind == "" {
|
||||
return "", errorf(http.StatusNotFound, "unknown application resource %q", plural)
|
||||
}
|
||||
return kind, nil
|
||||
}
|
||||
|
||||
// applicationStreamFilter selects the streams Fluent Bit labels with the
|
||||
// pod's lineage. vlselect ANDs each extra_stream_filters value into the
|
||||
// query and every subquery (Query.AddExtraFilters in lib/logstorage), so
|
||||
// it narrows the namespace filter sent next to it.
|
||||
func applicationStreamFilter(kind, name string) string {
|
||||
return "{apps_cozystack_io_application_kind=" + strconv.Quote(kind) + ",apps_cozystack_io_application_name=" + strconv.Quote(name) + "}"
|
||||
}
|
||||
|
||||
// post sends params as a form with only the given headers, so nothing the
|
||||
// client sent reaches the store, and tells an unreachable store (502) from
|
||||
// one that missed its deadline or, for a tail, its idle timeout (504).
|
||||
@@ -416,13 +451,15 @@ func (s *Server) authorize(r *http.Request, route Route) (string, *HTTPError) {
|
||||
slog.Error("token review failed", "error", err)
|
||||
return "", errorf(http.StatusServiceUnavailable, "authentication unavailable")
|
||||
}
|
||||
allowed, err := s.auth.Authorize(ctx, user, route.ResourceAttributes())
|
||||
if err != nil {
|
||||
slog.Error("subject access review failed", "error", err)
|
||||
return "", errorf(http.StatusServiceUnavailable, "authorization unavailable")
|
||||
}
|
||||
if !allowed {
|
||||
return "", errorf(http.StatusForbidden, "forbidden")
|
||||
for _, attrs := range route.Reviews() {
|
||||
allowed, err := s.auth.Authorize(ctx, user, attrs)
|
||||
if err != nil {
|
||||
slog.Error("subject access review failed", "error", err)
|
||||
return "", errorf(http.StatusServiceUnavailable, "authorization unavailable")
|
||||
}
|
||||
if !allowed {
|
||||
return "", errorf(http.StatusForbidden, "forbidden")
|
||||
}
|
||||
}
|
||||
return user.Username, nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,668 @@
|
||||
/*
|
||||
Copyright 2026 The Cozystack Authors.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package observabilityproxy
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
authorizationv1 "k8s.io/api/authorization/v1"
|
||||
)
|
||||
|
||||
const appRoot = "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/"
|
||||
|
||||
// appPolicy lets alice read every postgres in tenant-a and the namespace
|
||||
// roots, as the view role does. bob and carol may read only postgres "a",
|
||||
// as a grant narrowed with resourceNames does; carol may also read the
|
||||
// namespace roots.
|
||||
func appPolicy(user string, a *authorizationv1.ResourceAttributes) bool {
|
||||
if a.Namespace != "tenant-a" {
|
||||
return false
|
||||
}
|
||||
if a.Verb == "list" && a.Group == "cozystack.io" && (a.Resource == "logs" || a.Resource == "metrics") && a.Name == "" && a.Subresource == "" {
|
||||
return user == "alice" || user == "carol"
|
||||
}
|
||||
if a.Verb != "get" || a.Group != "apps.cozystack.io" || a.Resource != "postgreses" ||
|
||||
(a.Subresource != "log" && a.Subresource != "metrics") {
|
||||
return false
|
||||
}
|
||||
return user == "alice" || a.Name == "a"
|
||||
}
|
||||
|
||||
func TestServerApplicationLogsScope(t *testing.T) {
|
||||
h := newPolicyHarness(t, Config{}, nil, appPolicy, nil)
|
||||
hostile := url.Values{
|
||||
"query": {"error"},
|
||||
"field": {"level"},
|
||||
"extra_filters": {`{kubernetes_namespace_name="tenant-b"}`},
|
||||
"extra_stream_filters": {`{apps_cozystack_io_application_name=~".*"}`},
|
||||
"AccountID": {"9"},
|
||||
}
|
||||
headers := map[string]string{"AccountID": "9", "ProjectID": "9"}
|
||||
endpoints := []string{"query", "hits", "stats_query_range", "tail", "field_values", "stream_field_values", "facets"}
|
||||
for _, ep := range endpoints {
|
||||
resp := h.do(http.MethodPost, appRoot+"db.main/log/select/logsql/"+ep+"?extra_stream_filters=x", "good", hostile, headers)
|
||||
_, _ = io.Copy(io.Discard, resp.Body)
|
||||
_ = resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("%s answered %d", ep, resp.StatusCode)
|
||||
}
|
||||
}
|
||||
want := authorizationv1.ResourceAttributes{Verb: "get", Group: "apps.cozystack.io", Resource: "postgreses", Subresource: "log", Name: "db.main", Namespace: "tenant-a"}
|
||||
if got := h.kube.lastSAR.Load().ResourceAttributes; *got != want {
|
||||
t.Fatalf("SubjectAccessReview = %+v, want %+v", *got, want)
|
||||
}
|
||||
reqs := h.requests()
|
||||
if len(reqs) != len(endpoints) {
|
||||
t.Fatalf("upstream saw %d requests, want %d", len(reqs), len(endpoints))
|
||||
}
|
||||
// Two values: vlselect ANDs every extra_stream_filters value into the
|
||||
// query and each of its subqueries.
|
||||
wantFilters := []string{wantScope, `{apps_cozystack_io_application_kind="Postgres",apps_cozystack_io_application_name="db.main"}`}
|
||||
for i, r := range reqs {
|
||||
if r.path != "/select/logsql/"+endpoints[i] {
|
||||
t.Errorf("upstream path = %s, want %s", r.path, endpoints[i])
|
||||
}
|
||||
if got := r.form["extra_stream_filters"]; !slices.Equal(got, wantFilters) {
|
||||
t.Errorf("%s: extra_stream_filters = %q, want %q", endpoints[i], got, wantFilters)
|
||||
}
|
||||
if r.form.Has("extra_filters") || r.form.Has("AccountID") || r.headers.Get("AccountID") != "0" || r.headers.Get("ProjectID") != "0" {
|
||||
t.Errorf("%s: client scope reached the store: %v %v", endpoints[i], r.form, r.headers)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestServerApplicationKinds(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
kinds KindLookup
|
||||
path string
|
||||
want int
|
||||
}{
|
||||
{"unknown plural logs", nil, "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/mysqls/a/log/select/logsql/query", http.StatusNotFound},
|
||||
{"unknown plural metrics", nil, "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/mysqls/a/metrics/api/v1/query?query=up", http.StatusNotFound},
|
||||
{"not synced logs", newTestKinds(t, false), appRoot + "a/log/select/logsql/query", http.StatusServiceUnavailable},
|
||||
{"not synced metrics", newTestKinds(t, false), appRoot + "a/metrics/api/v1/query?query=up", http.StatusServiceUnavailable},
|
||||
{"ambiguous plural", newTestKinds(t, true, appDefinition("x", "Postgres", "postgreses"), appDefinition("y", "Pg", "postgreses")), appRoot + "a/log/select/logsql/query", http.StatusServiceUnavailable},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
h := newPolicyHarness(t, Config{}, nil, func(string, *authorizationv1.ResourceAttributes) bool { return true }, tt.kinds)
|
||||
if got := h.status(http.MethodGet, tt.path, "good"); got != tt.want {
|
||||
t.Fatalf("status = %d, want %d", got, tt.want)
|
||||
}
|
||||
if n := len(h.requests()); n != 0 {
|
||||
t.Fatalf("upstream saw %d requests", n)
|
||||
}
|
||||
})
|
||||
}
|
||||
// The kind is looked up only for an authorized caller.
|
||||
h := newPolicyHarness(t, Config{}, nil, appPolicy, newTestKinds(t, false))
|
||||
if got := h.status(http.MethodGet, appRoot+"b/log/select/logsql/query", "bob-token"); got != http.StatusForbidden {
|
||||
t.Fatalf("denied caller answered %d, want 403", got)
|
||||
}
|
||||
}
|
||||
|
||||
const wantPodLookup = `kube_pod_labels{namespace="tenant-a",label_apps_cozystack_io_application_kind="Postgres",label_apps_cozystack_io_application_name="db.main"}`
|
||||
|
||||
// appMetricsUpstream answers the pod and PVC lookups with the given names,
|
||||
// or with status when it is not 200, and every other path like vmselect.
|
||||
func appMetricsUpstream(pods, pvcs []string, podStatus int) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
names := map[string][]string{"/api/v1/label/pod/values": pods, "/api/v1/label/persistentvolumeclaim/values": pvcs}
|
||||
list, lookup := names[r.URL.Path]
|
||||
if !lookup {
|
||||
promUpstream(w, r)
|
||||
return
|
||||
}
|
||||
if r.URL.Path == "/api/v1/label/pod/values" && podStatus != http.StatusOK {
|
||||
http.Error(w, "unavailable", podStatus)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"status": "success", "data": list})
|
||||
}
|
||||
}
|
||||
|
||||
func TestServerApplicationMetricsScope(t *testing.T) {
|
||||
h := newPolicyHarness(t, Config{}, appMetricsUpstream([]string{"db-main-0", "db.main-1"}, []string{"data-db-main-0"}, http.StatusOK), appPolicy, nil)
|
||||
hostile := url.Values{
|
||||
"query": {"sum(rate(container_cpu_usage_seconds_total[5m]))"},
|
||||
"start": {unix(testNow.Add(-time.Hour))},
|
||||
"end": {unix(testNow)},
|
||||
"step": {"60"},
|
||||
"extra_filters": {`{namespace=~".+"}`},
|
||||
"extra_filters[]": {`{namespace="tenant-a"}`},
|
||||
"extra_label": {"apps_cozystack_io_application_name=b"},
|
||||
}
|
||||
resp := h.do(http.MethodPost, appRoot+"db.main/metrics/api/v1/query_range", "good", hostile, nil)
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
_ = resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("answered %d %q", resp.StatusCode, body)
|
||||
}
|
||||
want := authorizationv1.ResourceAttributes{Verb: "get", Group: "apps.cozystack.io", Resource: "postgreses", Subresource: "metrics", Name: "db.main", Namespace: "tenant-a"}
|
||||
if got := h.kube.lastSAR.Load().ResourceAttributes; *got != want {
|
||||
t.Fatalf("SubjectAccessReview = %+v, want %+v", *got, want)
|
||||
}
|
||||
reqs := h.requests()
|
||||
if len(reqs) != 3 {
|
||||
t.Fatalf("upstream saw %d requests, want two lookups and the query", len(reqs))
|
||||
}
|
||||
lookups := map[string]string{
|
||||
"/api/v1/label/pod/values": wantPodLookup,
|
||||
"/api/v1/label/persistentvolumeclaim/values": `kube_persistentvolumeclaim_labels{namespace="tenant-a",label_apps_cozystack_io_application_kind="Postgres",label_apps_cozystack_io_application_name="db.main"}`,
|
||||
}
|
||||
for _, r := range reqs[:2] {
|
||||
if got := r.form["match[]"]; len(got) != 1 || got[0] != lookups[r.path] {
|
||||
t.Errorf("%s match[] = %q", r.path, got)
|
||||
}
|
||||
if got := r.form["extra_filters[]"]; !slices.Equal(got, []string{wantMetricsScope}) {
|
||||
t.Errorf("%s extra_filters[] = %q", r.path, got)
|
||||
}
|
||||
// The query's own lookback is known and shorter than the 24h cap.
|
||||
if want := formatTime(buildMetrics(t, "api/v1/query_range", hostile).start); r.form.Get("start") != want {
|
||||
t.Errorf("%s start = %s, want the query's earliest read %s", r.path, r.form.Get("start"), want)
|
||||
}
|
||||
// One step past end, which vmselect rounds up to a whole step, and
|
||||
// two steps for each "(" that may be an implicit subquery.
|
||||
if r.form.Get("end") != formatTime(testNow.Add(5*time.Minute)) || r.form.Get("limit") != "101" {
|
||||
t.Errorf("%s end/limit = %s/%s", r.path, r.form.Get("end"), r.form.Get("limit"))
|
||||
}
|
||||
}
|
||||
main := reqs[2]
|
||||
wantBranches := []string{
|
||||
wantIdentityBranch,
|
||||
`{namespace="tenant-a",pod=~"^(db-main-0|db\\.main-1)$"}`,
|
||||
`{namespace="tenant-a",persistentvolumeclaim=~"^(data-db-main-0)$"}`,
|
||||
}
|
||||
if main.path != "/api/v1/query_range" || !slices.Equal(main.form["extra_filters[]"], wantBranches) {
|
||||
t.Fatalf("query %s extra_filters[] = %q, want %q", main.path, main.form["extra_filters[]"], wantBranches)
|
||||
}
|
||||
if main.form.Has("extra_filters") || main.form.Has("extra_label") {
|
||||
t.Fatalf("client scope reached the store: %v", main.form)
|
||||
}
|
||||
if !slices.Equal(h.stores, []string{"shortterm", "shortterm", "shortterm"}) {
|
||||
t.Fatalf("stores = %q, want the lookups on the query's store", h.stores)
|
||||
}
|
||||
}
|
||||
|
||||
func TestServerApplicationMetricsBranches(t *testing.T) {
|
||||
podBranch := `{namespace="tenant-a",pod=~"^(p)$"}`
|
||||
tests := []struct {
|
||||
name string
|
||||
pods, pvcs []string
|
||||
podStatus int
|
||||
podBody string
|
||||
cfg Config
|
||||
path string
|
||||
wantCode int
|
||||
want []string
|
||||
}{
|
||||
{name: "no pods or pvcs", path: "api/v1/query?query=up", wantCode: http.StatusOK, want: []string{wantIdentityBranch}},
|
||||
// A failed lookup would leave the application's own series out of
|
||||
// a successful answer, so it fails the request instead.
|
||||
{name: "pod lookup fails", pods: []string{"p"}, pvcs: []string{"v"}, podStatus: http.StatusServiceUnavailable, path: "api/v1/query?query=up", wantCode: http.StatusBadGateway},
|
||||
{name: "pod lookup times out at the store", pods: []string{"p"}, podStatus: http.StatusGatewayTimeout, path: "api/v1/query?query=up", wantCode: http.StatusGatewayTimeout},
|
||||
{name: "pods over the cap", pods: []string{"p", "q"}, cfg: Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 1, MaxPVCs: 1}}},
|
||||
path: "api/v1/query?query=up", wantCode: http.StatusUnprocessableEntity},
|
||||
{name: "over the cap with an empty name", pods: []string{"", "p"}, cfg: Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 1, MaxPVCs: 1}}},
|
||||
path: "api/v1/query?query=up", wantCode: http.StatusUnprocessableEntity},
|
||||
{name: "pvcs over the cap", pods: []string{"p"}, pvcs: []string{"v", "w"}, cfg: Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 1, MaxPVCs: 1}}},
|
||||
path: "api/v1/query?query=up", wantCode: http.StatusUnprocessableEntity},
|
||||
{name: "labels get the branches as match[]", pods: []string{"p"}, path: "api/v1/labels", wantCode: http.StatusOK, want: []string{wantIdentityBranch, podBranch}},
|
||||
{name: "series needs match[]", path: "api/v1/series", wantCode: http.StatusBadRequest},
|
||||
// An empty name would make pod=~"^(|p)$", which matches every
|
||||
// series without a pod label.
|
||||
{name: "empty pod name dropped", pods: []string{"", "p"}, path: "api/v1/query?query=up", wantCode: http.StatusOK, want: []string{wantIdentityBranch, podBranch}},
|
||||
{name: "only empty pod names", pods: []string{""}, path: "api/v1/query?query=up", wantCode: http.StatusOK, want: []string{wantIdentityBranch}},
|
||||
{name: "lookup status error", podBody: `{"status":"error","data":["p"]}`, path: "api/v1/query?query=up", wantCode: http.StatusBadGateway},
|
||||
{name: "truncated lookup answer", podBody: `{"status":"success","data":["p"`, path: "api/v1/query?query=up", wantCode: http.StatusBadGateway},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if tt.podStatus == 0 {
|
||||
tt.podStatus = http.StatusOK
|
||||
}
|
||||
upstream := appMetricsUpstream(tt.pods, tt.pvcs, tt.podStatus)
|
||||
if tt.podBody != "" {
|
||||
upstream = func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/api/v1/label/pod/values" {
|
||||
_, _ = io.WriteString(w, tt.podBody)
|
||||
return
|
||||
}
|
||||
appMetricsUpstream(nil, nil, http.StatusOK)(w, r)
|
||||
}
|
||||
}
|
||||
h := newPolicyHarness(t, tt.cfg, upstream, appPolicy, nil)
|
||||
if got := h.status(http.MethodGet, appRoot+"db.main/metrics/"+tt.path, "good"); got != tt.wantCode {
|
||||
t.Fatalf("status = %d, want %d", got, tt.wantCode)
|
||||
}
|
||||
var main *upstreamRequest
|
||||
for _, r := range h.requests() {
|
||||
if !strings.HasPrefix(r.path, "/api/v1/label/pod/") && !strings.HasPrefix(r.path, "/api/v1/label/persistentvolumeclaim/") {
|
||||
main = &r
|
||||
}
|
||||
}
|
||||
if tt.want == nil {
|
||||
if main != nil {
|
||||
t.Fatalf("query reached the store: %s %v", main.path, main.form)
|
||||
}
|
||||
return
|
||||
}
|
||||
if main == nil || !slices.Equal(main.form["extra_filters[]"], tt.want) {
|
||||
t.Fatalf("extra_filters[] = %v, want %q", main, tt.want)
|
||||
}
|
||||
if strings.HasSuffix(tt.path, "labels") && !slices.Equal(main.form["match[]"], tt.want) {
|
||||
t.Fatalf("match[] = %q, want the branches", main.form["match[]"])
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A grant narrowed to application "a" by resourceNames must not reach "b"
|
||||
// through any endpoint of either root. Without read access to the
|
||||
// namespace roots it does not reach "a" either: an ownerless pod keeps the
|
||||
// lineage labels its creator set, so it can claim to be "a" on both roots,
|
||||
// and on the metrics root the pod and PVC branches match by name, so a name
|
||||
// another application used earlier in the window would pull its series in.
|
||||
func TestServerApplicationNarrowedGrant(t *testing.T) {
|
||||
h := newPolicyHarness(t, Config{}, appMetricsUpstream([]string{"a-0"}, nil, http.StatusOK), appPolicy, nil)
|
||||
paths := []string{
|
||||
"metrics/api/v1/query?query=up",
|
||||
"metrics/api/v1/query_range?query=up&start=" + unix(testNow.Add(-time.Hour)) + "&end=" + unix(testNow) + "&step=60",
|
||||
"metrics/api/v1/series?match[]=up",
|
||||
"metrics/api/v1/labels",
|
||||
"metrics/api/v1/label/pod/values",
|
||||
"log/select/logsql/query",
|
||||
"log/select/logsql/hits",
|
||||
"log/select/logsql/stats_query_range?query=" + url.QueryEscape("* | stats count()"),
|
||||
"log/select/logsql/tail",
|
||||
"log/select/logsql/field_values?field=level",
|
||||
"log/select/logsql/stream_field_values?field=level",
|
||||
"log/select/logsql/facets",
|
||||
}
|
||||
for _, p := range paths {
|
||||
for _, denied := range []struct{ app, token string }{{"b", "bob-token"}, {"b", "carol-token"}, {"a", "bob-token"}} {
|
||||
if got := h.status(http.MethodGet, appRoot+denied.app+"/"+p, denied.token); got != http.StatusForbidden {
|
||||
t.Errorf("%s on %s/%s answered %d, want 403", denied.token, denied.app, p, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
if n := len(h.requests()); n != 0 {
|
||||
t.Fatalf("upstream saw %d requests for a denied application", n)
|
||||
}
|
||||
for _, p := range []string{"metrics/api/v1/labels", "log/select/logsql/query"} {
|
||||
if got := h.status(http.MethodGet, appRoot+"a/"+p, "carol-token"); got != http.StatusOK {
|
||||
t.Errorf("a/%s answered %d, want 200", p, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestServerApplicationLookbackBeyondCap: the lookup reaches back at most
|
||||
// --metrics-app-lookback, so a query that reads further back, or whose
|
||||
// lookback is unknown, would miss pods that existed only earlier; it is
|
||||
// refused rather than answered without their series.
|
||||
func TestServerApplicationLookbackBeyondCap(t *testing.T) {
|
||||
cfg := Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 100, MaxPVCs: 100}}}
|
||||
h := newPolicyHarness(t, cfg, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil)
|
||||
// max_over_time(x[2h]) reads just past the 1h cap; rate(x[5i]) has an
|
||||
// unknown lookback.
|
||||
for _, query := range []string{"rate(x[7d])", "max_over_time(x[2h])", "rate(x[5i])"} {
|
||||
q := url.Values{"query": {query}, "time": {unix(testNow)}}
|
||||
if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good"); got != http.StatusUnprocessableEntity {
|
||||
t.Errorf("%s answered %d, want 422", query, got)
|
||||
}
|
||||
}
|
||||
if n := len(h.requests()); n != 0 {
|
||||
t.Fatalf("upstream saw %d requests for refused queries", n)
|
||||
}
|
||||
q := url.Values{"query": {"rate(x[5m])"}, "time": {unix(testNow)}}
|
||||
if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good"); got != http.StatusOK {
|
||||
t.Fatalf("rate(x[5m]) answered %d, want 200", got)
|
||||
}
|
||||
r := h.requests()[0]
|
||||
want := formatTime(buildMetrics(t, "api/v1/query", q).start)
|
||||
if r.path != "/api/v1/label/pod/values" || r.form.Get("start") != want {
|
||||
t.Fatalf("lookup %s start = %s, want the query's earliest read %s", r.path, r.form.Get("start"), want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestServerApplicationLookupCoversNegativeOffset evaluates two hours ago
|
||||
// a selector shifted an hour forward: the pod lookup must reach the time
|
||||
// of the samples, or a pod created in between is left out.
|
||||
func TestServerApplicationLookupCoversNegativeOffset(t *testing.T) {
|
||||
h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil)
|
||||
at := testNow.Add(-2 * time.Hour)
|
||||
q := url.Values{"query": {"container_cpu_usage_seconds_total offset -1h"}, "time": {unix(at)}}
|
||||
if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good"); got != http.StatusOK {
|
||||
t.Fatalf("status = %d", got)
|
||||
}
|
||||
r := h.requests()[0]
|
||||
if want := formatTime(at.Add(time.Hour)); r.path != "/api/v1/label/pod/values" || r.form.Get("end") != want {
|
||||
t.Fatalf("lookup %s end = %s, want %s", r.path, r.form.Get("end"), want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestServerApplicationForwardsDefaultTime: without time the lookup is
|
||||
// sized at the proxy's clock, so that instant is forwarded. Otherwise
|
||||
// vmselect evaluates at its own later clock, past the end of the lookup by
|
||||
// however long the lookups took, and leaves out pods first sampled since.
|
||||
func TestServerApplicationForwardsDefaultTime(t *testing.T) {
|
||||
h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil)
|
||||
q := url.Values{"query": {"container_cpu_usage_seconds_total"}}
|
||||
if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good"); got != http.StatusOK {
|
||||
t.Fatalf("status = %d", got)
|
||||
}
|
||||
reqs := h.requests()
|
||||
lookup, final := reqs[0], reqs[len(reqs)-1]
|
||||
want := formatTime(testNow)
|
||||
if lookup.path != "/api/v1/label/pod/values" || lookup.form.Get("end") != want {
|
||||
t.Fatalf("lookup %s end = %s, want %s", lookup.path, lookup.form.Get("end"), want)
|
||||
}
|
||||
if final.path != "/api/v1/query" || final.form.Get("time") != want {
|
||||
t.Fatalf("query %s time = %q, want the lookup's instant %s", final.path, final.form.Get("time"), want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestServerApplicationRefusesUnboundedSampleTime refuses a query whose
|
||||
// samples may lie at a time the lookup cannot be sized for, rather than
|
||||
// answer with series silently left out.
|
||||
func TestServerApplicationRefusesUnboundedSampleTime(t *testing.T) {
|
||||
h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil)
|
||||
for _, query := range []string{"up @ 1700000000", "up offset -5i"} {
|
||||
q := url.Values{"query": {query}}
|
||||
if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good"); got != http.StatusUnprocessableEntity {
|
||||
t.Errorf("%s answered %d, want 422", query, got)
|
||||
}
|
||||
}
|
||||
if n := len(h.requests()); n != 0 {
|
||||
t.Fatalf("upstream saw %d requests for a refused query", n)
|
||||
}
|
||||
// The namespace root has no lookup to size and still serves it.
|
||||
if got := h.status(http.MethodGet, metricsRoot+"api/v1/query?"+url.Values{"query": {"up @ 1700000000"}}.Encode(), "good"); got != http.StatusOK {
|
||||
t.Fatalf("namespace root answered %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestServerApplicationLookupCoversAlignedEnd: vmselect rounds a range
|
||||
// query's end up to a whole step, so the lookup reaches one step past it.
|
||||
func TestServerApplicationLookupCoversAlignedEnd(t *testing.T) {
|
||||
h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil)
|
||||
end := testNow.Add(-90 * time.Minute)
|
||||
q := url.Values{"query": {"up"}, "start": {unix(end.Add(-6 * time.Hour))}, "end": {unix(end)}, "step": {"3600"}}
|
||||
if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query_range?"+q.Encode(), "good"); got != http.StatusOK {
|
||||
t.Fatalf("status = %d", got)
|
||||
}
|
||||
r := h.requests()[0]
|
||||
if want := formatTime(end.Add(time.Hour)); r.path != "/api/v1/label/pod/values" || r.form.Get("end") != want {
|
||||
t.Fatalf("lookup %s end = %s, want %s", r.path, r.form.Get("end"), want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestServerApplicationLookupDeadline: a lookup that misses the request
|
||||
// deadline answers 504, whether the store sent nothing or stalled after
|
||||
// its headers, and the query itself is not sent.
|
||||
func TestServerApplicationLookupDeadline(t *testing.T) {
|
||||
for _, headers := range []bool{false, true} {
|
||||
h := newPolicyHarness(t, Config{RequestTimeout: 50 * time.Millisecond}, func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/api/v1/label/pod/values" {
|
||||
appMetricsUpstream(nil, nil, http.StatusOK)(w, r)
|
||||
return
|
||||
}
|
||||
if headers {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = io.WriteString(w, `{"status":"success","data":[`)
|
||||
w.(http.Flusher).Flush()
|
||||
}
|
||||
<-r.Context().Done()
|
||||
}, appPolicy, nil)
|
||||
if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?query=up", "good"); got != http.StatusGatewayTimeout {
|
||||
t.Fatalf("headers sent %v: status = %d, want 504", headers, got)
|
||||
}
|
||||
for _, r := range h.requests() {
|
||||
if !strings.HasPrefix(r.path, "/api/v1/label/") {
|
||||
t.Fatalf("headers sent %v: the query reached the store after a failed lookup", headers)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestServerApplicationDefaultLookbackServesLongPanels: the lookback is
|
||||
// only the refusal threshold, and the lookback estimate is a loose upper
|
||||
// bound, so the default must serve a day's increase and a month-long
|
||||
// panel at the step a dashboard picks for it.
|
||||
func TestServerApplicationDefaultLookbackServesLongPanels(t *testing.T) {
|
||||
cfg := Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: 30 * 24 * time.Hour, MaxPods: 100, MaxPVCs: 100}}}
|
||||
h := newPolicyHarness(t, cfg, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil)
|
||||
month := url.Values{"start": {unix(testNow.Add(-29 * 24 * time.Hour))}, "end": {unix(testNow)}}
|
||||
for _, tc := range []struct{ path, query, step string }{
|
||||
{"api/v1/query", "increase(x[24h])", ""},
|
||||
{"api/v1/query_range", "rate(x[5m])", "21600"},
|
||||
{"api/v1/query_range", "histogram_quantile(0.99, sum by (le) (rate(x[5m])))", "7200"},
|
||||
} {
|
||||
q := url.Values{"query": {tc.query}}
|
||||
if tc.step != "" {
|
||||
q.Set("step", tc.step)
|
||||
q.Set("start", month.Get("start"))
|
||||
q.Set("end", month.Get("end"))
|
||||
}
|
||||
if got := h.status(http.MethodGet, appRoot+"db.main/metrics/"+tc.path+"?"+q.Encode(), "good"); got != http.StatusOK {
|
||||
t.Errorf("%s step %s answered %d, want 200", tc.query, tc.step, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestServerApplicationLookupStoreErrors: vmselect answers 422 to any error
|
||||
// without a status of its own (SendPrometheusError), so only its series
|
||||
// limit, which a narrower window fixes, reaches the client as 422.
|
||||
func TestServerApplicationLookupStoreErrors(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
label, message string
|
||||
want int
|
||||
}{
|
||||
{"pod", "the number of matching timeseries exceeds 300000; either narrow down the search or increase -search.max* command-line flag values", http.StatusUnprocessableEntity},
|
||||
{"persistentvolumeclaim", "the number of matching timeseries exceeds 300000; either narrow down the search", http.StatusUnprocessableEntity},
|
||||
{"pod", "cannot fetch label values from vmstorage nodes: the number of matching timeseries exceeds 300000", http.StatusUnprocessableEntity},
|
||||
{"pod", "timeout exceeded before starting the query processing: 30s", http.StatusGatewayTimeout},
|
||||
{"pod", "error when searching for label values: cannot obtain data from vmstorage", http.StatusBadGateway},
|
||||
} {
|
||||
h := newPolicyHarness(t, Config{}, func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/api/v1/label/"+tc.label+"/values" {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusUnprocessableEntity)
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"status": "error", "errorType": "422", "error": tc.message})
|
||||
return
|
||||
}
|
||||
appMetricsUpstream([]string{"p"}, nil, http.StatusOK)(w, r)
|
||||
}, appPolicy, nil)
|
||||
resp := h.do(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?query=up", "good", nil, nil)
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
_ = resp.Body.Close()
|
||||
if resp.StatusCode != tc.want {
|
||||
t.Errorf("%s lookup %q answered %d, want %d", tc.label, tc.message, resp.StatusCode, tc.want)
|
||||
}
|
||||
if tc.want == http.StatusUnprocessableEntity && !strings.Contains(string(body), "the number of matching timeseries exceeds 300000") {
|
||||
t.Errorf("the store's limit message did not reach the client: %q", body)
|
||||
}
|
||||
if strings.Contains(string(body), `"errorType"`) {
|
||||
t.Errorf("the store's raw JSON reached the client: %q", body)
|
||||
}
|
||||
// Other store errors may name vmstorage nodes; only the limit
|
||||
// message, which the client can act on, is passed on.
|
||||
if strings.Contains(string(body), "vmstorage") {
|
||||
t.Errorf("the store's error text reached the client: %q", body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestServerApplicationRefusesPartialLookup: vmselect answers a label
|
||||
// values lookup with isPartial when a vmstorage node is down, and a short
|
||||
// pod list would drop the missing pods' series without a word.
|
||||
func TestServerApplicationRefusesPartialLookup(t *testing.T) {
|
||||
for _, label := range []string{"pod", "persistentvolumeclaim"} {
|
||||
h := newPolicyHarness(t, Config{}, func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/api/v1/label/"+label+"/values" {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = io.WriteString(w, `{"status":"success","isPartial":true,"data":["db-0"]}`)
|
||||
return
|
||||
}
|
||||
appMetricsUpstream([]string{"p"}, nil, http.StatusOK)(w, r)
|
||||
}, appPolicy, nil)
|
||||
if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?query=up", "good"); got != http.StatusBadGateway {
|
||||
t.Errorf("partial %s lookup answered %d, want 502", label, got)
|
||||
}
|
||||
for _, req := range h.requests() {
|
||||
if req.path == "/api/v1/query" {
|
||||
t.Errorf("partial %s lookup still sent the query", label)
|
||||
}
|
||||
}
|
||||
}
|
||||
// A partial list already over the cap settles it: the full one is longer.
|
||||
cfg := Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 1, MaxPVCs: 1}}}
|
||||
h := newPolicyHarness(t, cfg, func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/api/v1/label/pod/values" {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = io.WriteString(w, `{"status":"success","isPartial":true,"data":["db-0","db-1"]}`)
|
||||
return
|
||||
}
|
||||
appMetricsUpstream([]string{"p"}, nil, http.StatusOK)(w, r)
|
||||
}, appPolicy, nil)
|
||||
if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?query=up", "good"); got != http.StatusUnprocessableEntity {
|
||||
t.Errorf("partial pod list over the cap answered %d, want 422", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestServerApplicationRefusesEveryAt: no @ form can be bounded, and the
|
||||
// answer says so rather than naming forms that would pass.
|
||||
func TestServerApplicationRefusesEveryAt(t *testing.T) {
|
||||
h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil)
|
||||
for _, query := range []string{"up @ end()", "up @ start()", "up @ 1700000000"} {
|
||||
resp := h.do(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+url.Values{"query": {query}}.Encode(), "good", nil, nil)
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
_ = resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusUnprocessableEntity {
|
||||
t.Errorf("%s answered %d, want 422", query, resp.StatusCode)
|
||||
}
|
||||
if strings.Contains(string(body), "start() or end()") {
|
||||
t.Errorf("%s: the refusal names @ forms as allowed: %q", query, body)
|
||||
}
|
||||
}
|
||||
if n := len(h.requests()); n != 0 {
|
||||
t.Fatalf("upstream saw %d requests for refused queries", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestServerApplicationRefusesForwardReads: queries that read past the
|
||||
// evaluation time by an amount the scan cannot bound answer 422 and send
|
||||
// nothing upstream, rather than answer 200 without pods created in that
|
||||
// time.
|
||||
func TestServerApplicationRefusesForwardReads(t *testing.T) {
|
||||
h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil)
|
||||
for _, query := range []string{
|
||||
"container_cpu_usage_seconds_total offset -1h-30m",
|
||||
"container_cpu_usage_seconds_total offset 1h-2h",
|
||||
"container_cpu_usage_seconds_total offset -1h$__interval",
|
||||
"rollup_candlestick(container_cpu_usage_seconds_total)",
|
||||
} {
|
||||
q := url.Values{"query": {query}, "time": {unix(testNow.Add(-2 * time.Hour))}}
|
||||
resp := h.do(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good", nil, nil)
|
||||
_ = resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusUnprocessableEntity {
|
||||
t.Errorf("%s answered %d, want 422", query, resp.StatusCode)
|
||||
}
|
||||
}
|
||||
if n := len(h.requests()); n != 0 {
|
||||
t.Fatalf("upstream saw %d requests for refused queries", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestServerApplicationLookupCoversSubquery: a pod created during the two
|
||||
// subquery steps past the evaluation time can show up through range_last,
|
||||
// so the lookup reaches that far.
|
||||
func TestServerApplicationLookupCoversSubquery(t *testing.T) {
|
||||
h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil)
|
||||
at := testNow.Add(-6 * time.Hour)
|
||||
q := url.Values{"query": {"max_over_time(range_last(container_cpu_usage_seconds_total)[1h:1h])"}, "time": {unix(at)}}
|
||||
if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good"); got != http.StatusOK {
|
||||
t.Fatalf("status = %d", got)
|
||||
}
|
||||
r := h.requests()[0]
|
||||
end, err := time.Parse(time.RFC3339, r.form.Get("end"))
|
||||
if err != nil || r.path != "/api/v1/label/pod/values" || end.Before(at.Add(2*time.Hour)) {
|
||||
t.Fatalf("lookup %s end = %s, want at least %s", r.path, r.form.Get("end"), formatTime(at.Add(2*time.Hour)))
|
||||
}
|
||||
}
|
||||
|
||||
// TestServerApplicationNamesUnreadableSubqueryStep: the refusal names the
|
||||
// cause when a subquery step is what the scan cannot read.
|
||||
func TestServerApplicationNamesUnreadableSubqueryStep(t *testing.T) {
|
||||
h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil)
|
||||
resp := h.do(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+url.Values{"query": {"max_over_time(x[1h:5i])"}}.Encode(), "good", nil, nil)
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
_ = resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusUnprocessableEntity || !strings.Contains(string(body), "subquery step") {
|
||||
t.Fatalf("answered %d %q, want 422 naming the subquery step", resp.StatusCode, body)
|
||||
}
|
||||
}
|
||||
|
||||
// TestServerApplicationLookupCoversLatencyOffset: vmselect evaluates an
|
||||
// instant query whose time is within 30s of its own clock at its clock
|
||||
// minus 30s (prometheus.go, queryOffset), so for a time slightly in the
|
||||
// future it reads from earlier than time minus 30s. The lookup starts
|
||||
// early enough for any evaluation instant in (time-60s, time].
|
||||
func TestServerApplicationLookupCoversLatencyOffset(t *testing.T) {
|
||||
h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil)
|
||||
at := testNow.Add(29 * time.Second)
|
||||
q := url.Values{"query": {"container_cpu_usage_seconds_total"}, "time": {at.Format(time.RFC3339)}, "step": {"1s"}}
|
||||
if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good"); got != http.StatusOK {
|
||||
t.Fatalf("status = %d", got)
|
||||
}
|
||||
// The earliest read of the query evaluated at the earliest instant.
|
||||
lookback, _ := queryLookback("container_cpu_usage_seconds_total", time.Second)
|
||||
want := at.Add(-2*latencyOffset - lookback)
|
||||
r := h.requests()[0]
|
||||
start, err := time.Parse(time.RFC3339, r.form.Get("start"))
|
||||
if err != nil || start.After(want) {
|
||||
t.Fatalf("lookup start = %s, want at most %s", r.form.Get("start"), formatTime(want))
|
||||
}
|
||||
}
|
||||
|
||||
// TestServerApplicationRefusesBackquoteEscapeTail: an escaped backquote
|
||||
// inside a quoted label name does not end the string in metricsql, so the
|
||||
// escaped rollup_candlestick after it must still be refused.
|
||||
func TestServerApplicationRefusesBackquoteEscapeTail(t *testing.T) {
|
||||
h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil)
|
||||
for _, query := range []string{
|
||||
"container_cpu_usage_seconds_total{`\\``=\"\"} or rollup\\_candlestick(container_cpu_usage_seconds_total offset -5m+1h)",
|
||||
"{`\\`a`} or rollup\\_candlestick(container_cpu_usage_seconds_total offset -5m+1h)",
|
||||
} {
|
||||
q := url.Values{"query": {query}, "time": {unix(testNow.Add(-6 * time.Hour))}}
|
||||
if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good"); got != http.StatusUnprocessableEntity {
|
||||
t.Errorf("%s answered %d, want 422", query, got)
|
||||
}
|
||||
}
|
||||
if n := len(h.requests()); n != 0 {
|
||||
t.Fatalf("upstream saw %d requests for refused queries", n)
|
||||
}
|
||||
}
|
||||
@@ -428,6 +428,8 @@ func TestServerStalledReviewAnswersInTime(t *testing.T) {
|
||||
{"logs subject access review", logsRoot + "query", false},
|
||||
{"metrics token review", metricsRoot + "api/v1/query", true},
|
||||
{"metrics subject access review", metricsRoot + "api/v1/query", false},
|
||||
{"application logs subject access review", appRoot + "db/log/select/logsql/query", false},
|
||||
{"application metrics subject access review", appRoot + "db/metrics/api/v1/query", false},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
h := newHarness(t, Config{RequestTimeout: 200 * time.Millisecond}, nil)
|
||||
|
||||
@@ -203,7 +203,7 @@ func TestServerMetricsConcurrency(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestDefaultMetricsUpstreamURL(t *testing.T) {
|
||||
s := NewServer(Config{ClusterDomain: "cozy.local", Metrics: MetricsConfig{RootTenant: "tenant-root"}}, nil, nil, nil)
|
||||
s := NewServer(Config{ClusterDomain: "cozy.local", Metrics: MetricsConfig{RootTenant: "tenant-root"}}, nil, nil, nil, nil)
|
||||
if got, want := s.cfg.Metrics.UpstreamURL("shortterm"), "http://vmselect-shortterm.tenant-root.svc.cozy.local:8481/select/0/prometheus"; got != want {
|
||||
t.Fatalf("upstream = %s, want %s", got, want)
|
||||
}
|
||||
|
||||
@@ -56,10 +56,19 @@ func (m mapStores) Resolve(ns string) (string, error) { return m[ns], nil }
|
||||
// (alice) and "bob-token" (bob), both allowed list logs and metrics in
|
||||
// tenant-a only, and to an upstream served by handler.
|
||||
func newHarness(t *testing.T, cfg Config, handler http.HandlerFunc) *harness {
|
||||
h := &harness{t: t}
|
||||
h.kube = newFakeKube(map[string]string{"good": "alice", "bob-token": "bob"}, func(_ string, a *authorizationv1.ResourceAttributes) bool {
|
||||
return newPolicyHarness(t, cfg, handler, func(_ string, a *authorizationv1.ResourceAttributes) bool {
|
||||
return a.Namespace == "tenant-a" && (a.Resource == "logs" || a.Resource == "metrics")
|
||||
})
|
||||
}, nil)
|
||||
}
|
||||
|
||||
// newPolicyHarness is newHarness with the given SubjectAccessReview policy
|
||||
// and plural-to-kind mapping; nil kinds knows only postgreses.
|
||||
func newPolicyHarness(t *testing.T, cfg Config, handler http.HandlerFunc, allow func(string, *authorizationv1.ResourceAttributes) bool, kinds KindLookup) *harness {
|
||||
h := &harness{t: t}
|
||||
h.kube = newFakeKube(map[string]string{"good": "alice", "bob-token": "bob", "carol-token": "carol"}, allow)
|
||||
if kinds == nil {
|
||||
kinds = newTestKinds(t, true, appDefinition("postgres", "Postgres", "postgreses"))
|
||||
}
|
||||
h.upstream = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if err := r.ParseForm(); err != nil {
|
||||
t.Errorf("upstream ParseForm: %v", err)
|
||||
@@ -88,10 +97,13 @@ func newHarness(t *testing.T, cfg Config, handler http.HandlerFunc) *harness {
|
||||
if cfg.Metrics.Limits == (MetricsLimits{}) {
|
||||
cfg.Metrics.Limits = testMetricsLimits()
|
||||
}
|
||||
if cfg.Metrics.App == (AppScopeLimits{}) {
|
||||
cfg.Metrics.App = AppScopeLimits{Lookback: 24 * time.Hour, MaxPods: 100, MaxPVCs: 100}
|
||||
}
|
||||
if cfg.Logs == (LogsLimits{}) {
|
||||
cfg.Logs = testLimits()
|
||||
}
|
||||
h.server = NewServer(cfg, NewAuthorizer(h.kube.client, 10*time.Second), mapStores{"tenant-a": "tenant-a-store"}, nil)
|
||||
h.server = NewServer(cfg, NewAuthorizer(h.kube.client, 10*time.Second), mapStores{"tenant-a": "tenant-a-store"}, kinds, nil)
|
||||
h.server.now = func() time.Time { return testNow }
|
||||
h.proxy = httptest.NewServer(h.server)
|
||||
t.Cleanup(h.proxy.Close)
|
||||
@@ -155,8 +167,8 @@ func TestServerStatusCodes(t *testing.T) {
|
||||
{"unknown metrics endpoint", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/metrics/api/v1/status/tsdb", "good", http.StatusNotFound},
|
||||
{"metrics in another namespace", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-b/metrics/api/v1/query?query=up", "good", http.StatusForbidden},
|
||||
{"metrics without token", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/metrics/api/v1/query?query=up", "", http.StatusUnauthorized},
|
||||
{"application log not served yet", http.MethodGet, "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/db/log/select/logsql/query", "good", http.StatusNotFound},
|
||||
{"application metrics not served yet", http.MethodGet, "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/db/metrics/api/v1/query", "good", http.StatusNotFound},
|
||||
{"application log without a grant", http.MethodGet, "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/db/log/select/logsql/query", "good", http.StatusForbidden},
|
||||
{"application metrics without a grant", http.MethodGet, "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/db/metrics/api/v1/query", "good", http.StatusForbidden},
|
||||
{"tenant_ids not proxied", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/logs/select/tenant_ids", "good", http.StatusNotFound},
|
||||
{"unknown logsql endpoint", http.MethodGet, logsRoot + "stats_query", "good", http.StatusNotFound},
|
||||
{"logs root itself", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/logs/", "good", http.StatusNotFound},
|
||||
|
||||
@@ -45,6 +45,7 @@ spec:
|
||||
- --metrics-shortterm-store={{ .Values.observabilityProxy.metrics.shorttermStore }}
|
||||
- --metrics-longterm-store={{ .Values.observabilityProxy.metrics.longtermStore }}
|
||||
- --metrics-shortterm-retention={{ .Values.observabilityProxy.metrics.shorttermRetention }}
|
||||
- --metrics-app-lookback={{ .Values.observabilityProxy.metrics.appLookback }}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8080
|
||||
|
||||
@@ -15,6 +15,11 @@ rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["namespaces"]
|
||||
verbs: ["list", "watch"]
|
||||
# Application roots map the plural in the path to the kind the lineage
|
||||
# labels carry, from an informer cache as well.
|
||||
- apiGroups: ["cozystack.io"]
|
||||
resources: ["applicationdefinitions"]
|
||||
verbs: ["list", "watch"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
|
||||
@@ -1,7 +1,11 @@
|
||||
# Aggregated into cozy-tenant-view (and consequently use, admin, super-admin).
|
||||
# No API server serves cozystack.io logs or metrics: the rule exists only for
|
||||
# the SubjectAccessReview the observability proxy asks before it answers a
|
||||
# read under /apis/cozystack.io/v1alpha1/namespaces/<ns>/{logs,metrics}/.
|
||||
# read under /apis/cozystack.io/v1alpha1/namespaces/<ns>/{logs,metrics}/,
|
||||
# and before a read under an application root,
|
||||
# /apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/{log,metrics}/,
|
||||
# which needs the namespace root too. Narrowing this role also cuts off the
|
||||
# application roots.
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
|
||||
@@ -60,6 +60,9 @@ tests:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].args
|
||||
content: --metrics-shortterm-retention=72h
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].args
|
||||
content: --metrics-app-lookback=720h
|
||||
|
||||
- it: follows renamed metrics stores
|
||||
set:
|
||||
@@ -67,6 +70,7 @@ tests:
|
||||
shorttermStore: fast
|
||||
longtermStore: slow
|
||||
shorttermRetention: 24h
|
||||
appLookback: 48h
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].args
|
||||
@@ -77,6 +81,9 @@ tests:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].args
|
||||
content: --metrics-shortterm-retention=24h
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].args
|
||||
content: --metrics-app-lookback=48h
|
||||
|
||||
- it: falls back to cozy.local when the platform passes no cluster domain
|
||||
asserts:
|
||||
|
||||
@@ -12,7 +12,7 @@ release:
|
||||
namespace: cozy-observability-proxy
|
||||
|
||||
tests:
|
||||
- it: lets the proxy review tokens and access, and list and watch namespaces
|
||||
- it: lets the proxy review tokens and access, and list and watch namespaces and application definitions
|
||||
template: templates/rbac.yaml
|
||||
documentSelector:
|
||||
path: kind
|
||||
@@ -30,6 +30,9 @@ tests:
|
||||
- apiGroups: [""]
|
||||
resources: ["namespaces"]
|
||||
verbs: ["list", "watch"]
|
||||
- apiGroups: ["cozystack.io"]
|
||||
resources: ["applicationdefinitions"]
|
||||
verbs: ["list", "watch"]
|
||||
|
||||
- it: binds the proxy role to the proxy ServiceAccount in the release namespace
|
||||
template: templates/rbac.yaml
|
||||
|
||||
@@ -14,6 +14,9 @@ observabilityProxy:
|
||||
shorttermStore: shortterm
|
||||
longtermStore: longterm
|
||||
shorttermRetention: 72h
|
||||
# How far before its window an application metrics query may read;
|
||||
# one reading further back answers 422 on the application root.
|
||||
appLookback: 720h
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
|
||||
Reference in new issue
Block a user