mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
feat(monitoring): stamp application identity on scraped series (#5216)
## What this PR does
Adds a default scrape class `application-identity` to both VMAgents: the
cluster-wide one in `monitoring-agents` and the per-tenant one in
`monitoring`. Every series scraped from a pod that carries the lineage
labels `apps.cozystack.io/application.{group,kind,name}` gets them as
`apps_cozystack_io_application_{group,kind,name}`.
Why: the per-application metrics scope in the design proposal [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94) (Design 4a)
needs to tell which application an exporter series belongs to. Today
that is only possible indirectly, through pod names.
How the labels are protected against forgery by an exporter:
- At the target stage, the class always sets all three labels: to the
pod's real lineage value, or to a placeholder `-` when the pod has none
(`-` is not a valid Kubernetes label value). With `honorLabels: false`,
any copy the exporter publishes therefore collides with the target label
and is renamed to `exported_*`.
- At the metric stage, the class drops every `exported_*` copy.
- The placeholder is removed by the VMAgent's global remote-write
relabeling (`inlineRelabelConfig`), not at the metric stage: vmagent
adds `up` and `scrape_*` with the target labels after metric relabeling,
so only the remote-write stage sees those series. Targets without
lineage therefore keep exactly the label set they have today on every
series, including `up` and `scrape_*`.
- With `honorLabels: true` the exporter's value wins and no relabel rule
can tell it apart. ingress-nginx scrapes with `honorLabels: true`, and
its series describe Ingress objects in any namespace, so the
controller's identity would misattribute traffic; its scrape now drops
the identity labels explicitly, from the target (so `up` and `scrape_*`
stay unchanged) and from the scraped samples (so an exporter's copy
cannot survive). Switching it to `honorLabels: false` was not done,
because that renames `namespace` and breaks the existing ingress
dashboards.
#### How it was tested
- helm-unittest for `monitoring-agents`, `monitoring` and `ingress`.
These tests check the rendered config only; the relabeling behaviour was
verified offline by running vmagent v1.136.0, built from source, against
the rendered rules: `up` and `scrape_*` carry no identity for targets
without lineage, and the placeholder appears on no series.
- The rendered `scrapeClasses` validate against the vendored VMAgent CRD
schema.
- The rendered rules were exercised on a locally built vmagent v1.136.0
(the version the bundled operator deploys) writing to a local
VictoriaMetrics: with and without lineage, with and without a forging
exporter, `honorLabels` true and false, and the ingress-nginx endpoint
rules. Checked on exporter samples and on `up`, `scrape_samples_scraped`
and `scrape_series_added`; the previous rules reproduce the `-`
placeholder on the generated series of targets without lineage, the new
ones leave no placeholder on any series.
- Verified on a test cluster with the earlier revision that removed the
placeholder at the metric stage (the move to remote-write relabeling is
not yet verified on a cluster): series carry the application labels;
metrics read for a Redis instance through the per-application proxy root
contained only that instance's pods; vmagent logs show no errors.
#### Known limitations
- Scrapes with `honorLabels: true` other than ingress-nginx are not
protected against an exporter that publishes the identity labels itself.
Today these are the vendored qdrant and harbor ServiceMonitors; their
exporters are platform images with fixed label names, and their series
describe the application's own pods.
- A tenant's raw `vmagent.inlineScrapeConfig` bypasses scrape classes
entirely, and so do `additionalScrapeConfigs`. VMProbe takes only auth
settings from a scrape class (operator v0.68.4); no VMProbe exists in
the platform today.
- A scrape object's own relabel rules run after the class and can
overwrite the labels; tenants cannot create VictoriaMetrics operator
objects, so this is platform-controlled.
### Screenshots
No UI changes.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
Downstream: no trigger matched. Candidate: none affected, because the
diff changes templates and tests in `packages/system/monitoring-agents`,
`packages/system/monitoring` and `packages/extra/ingress` without
adding, renaming or removing a package and without touching any
`values.yaml` or schema. The telemetry metrics the telemetry server
reads (`cozy_cluster_info`, `cozy_application_count`) are produced by
the operator, not by these scrapes. The proposal text in
`cozystack/community` (#94, Design 4a) should be corrected to name
qdrant and harbor as further `honorLabels: true` scrapes; that is not a
trigger-map item.
### Release note
```release-note
feat(monitoring): VMAgent now adds `apps_cozystack_io_application_{group,kind,name}` labels to every series scraped from a pod that carries application lineage labels. Because the label set of those series changes, each of them becomes a new series after the upgrade: graphs and dashboards show a break at the rollout, queries over a range that spans it see two series per pod, and alert `for:` timers on those series restart. Series of targets without lineage labels, including node-level series (cAdvisor, kubelet) and kube-state-metrics, and all ingress-nginx series keep their current label set; this includes their `up` and `scrape_*` series.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Improvements**
* Monitoring metrics now use application identity labels derived from
Kubernetes pod metadata for covered scrape targets. Exporter-supplied
duplicates and placeholder values are removed, including from `up` and
`scrape_*` metrics.
* Ingress metrics no longer retain application identity labels at either
target or metric relabeling stages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
6 files changed
+295
No files matched your search
@@ -9,6 +9,13 @@ spec:
|
||||
matchNames:
|
||||
- {{ .Release.Namespace }}
|
||||
podMetricsEndpoints:
|
||||
# Most series describe the Ingress objects served, in any namespace, so
|
||||
# the application identity the vmagent scrape class stamps from the
|
||||
# controller pod would misattribute them; and with honorLabels an
|
||||
# exporter-supplied identity would win over the class. The controller's
|
||||
# own series (nginx process, reload status, go_*, up, scrape_*) lose the
|
||||
# Ingress application's identity too; that is the accepted cost. The
|
||||
# target-stage drop covers up and scrape_*, which skip metric relabeling.
|
||||
- port: metrics
|
||||
honorLabels: true
|
||||
interval: 30s
|
||||
@@ -19,6 +26,11 @@ spec:
|
||||
targetLabel: node
|
||||
- targetLabel: tier
|
||||
replacement: cluster
|
||||
- action: labeldrop
|
||||
regex: apps_cozystack_io_application_(group|kind|name)
|
||||
metricRelabelConfigs:
|
||||
- action: labeldrop
|
||||
regex: apps_cozystack_io_application_(group|kind|name)
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/component: controller
|
||||
@@ -46,6 +58,11 @@ spec:
|
||||
targetLabel: controller
|
||||
- targetLabel: tier
|
||||
replacement: cluster
|
||||
- action: labeldrop
|
||||
regex: apps_cozystack_io_application_(group|kind|name)
|
||||
metricRelabelConfigs:
|
||||
- action: labeldrop
|
||||
regex: apps_cozystack_io_application_(group|kind|name)
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/component: controller
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
suite: ingress-nginx series do not carry an application identity
|
||||
templates:
|
||||
- templates/nginx-scrape.yaml
|
||||
release:
|
||||
name: ingress
|
||||
namespace: tenant-root
|
||||
|
||||
# The vmagent scrape class stamps the scraped pod's application identity on
|
||||
# every series. Most ingress-nginx series describe the Ingress objects it
|
||||
# serves, in whatever namespace they live (honorLabels keeps their
|
||||
# namespace), so the controller's identity would misattribute them, and with
|
||||
# honorLabels true an exporter-supplied identity would win over the class.
|
||||
# The controller's own health series lose that identity too, by design. Both
|
||||
# endpoints drop the identity instead: from the target, because vmagent
|
||||
# adds up and scrape_* with the target labels after metric relabeling, and
|
||||
# from the scraped samples, because an exporter's copy is not renamed.
|
||||
tests:
|
||||
- it: drops the identity on every endpoint
|
||||
asserts:
|
||||
- lengthEqual:
|
||||
path: spec.podMetricsEndpoints
|
||||
count: 1
|
||||
- equal:
|
||||
path: spec.podMetricsEndpoints[0].metricRelabelConfigs
|
||||
value:
|
||||
- action: labeldrop
|
||||
regex: apps_cozystack_io_application_(group|kind|name)
|
||||
|
||||
# Endpoint rules run after the scrape class rules, so the drop takes
|
||||
# effect wherever it sits in this list; it is pinned last so that no
|
||||
# later rule can set the identity again.
|
||||
- it: drops the identity from the basic endpoint's target as its last rule
|
||||
documentIndex: 0
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.podMetricsEndpoints[0].relabelConfigs
|
||||
value:
|
||||
- replacement: nginx-ingress-controller
|
||||
targetLabel: job
|
||||
- sourceLabels: [__meta_kubernetes_pod_node_name]
|
||||
targetLabel: node
|
||||
- targetLabel: tier
|
||||
replacement: cluster
|
||||
- action: labeldrop
|
||||
regex: apps_cozystack_io_application_(group|kind|name)
|
||||
|
||||
- it: drops the identity from the detailed endpoint's target as its last rule
|
||||
documentIndex: 1
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.podMetricsEndpoints[0].relabelConfigs
|
||||
value:
|
||||
- replacement: nginx-ingress-controller
|
||||
targetLabel: job
|
||||
- sourceLabels: [__meta_kubernetes_pod_node_name]
|
||||
targetLabel: node
|
||||
- sourceLabels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
|
||||
targetLabel: controller
|
||||
- targetLabel: tier
|
||||
replacement: cluster
|
||||
- action: labeldrop
|
||||
regex: apps_cozystack_io_application_(group|kind|name)
|
||||
@@ -20,6 +20,39 @@ spec:
|
||||
|
||||
scrapeInterval: 30s
|
||||
selectAllByDefault: true
|
||||
# The class sets all three identity labels on every target it covers, "-"
|
||||
# when the pod has no lineage ("-" is not a valid Kubernetes label value,
|
||||
# so it never collides with a real one). Under honorLabels false every
|
||||
# exporter-supplied copy then collides, is renamed to exported_* and
|
||||
# dropped. Under honorLabels true the exporter's copy wins, so such a
|
||||
# scrape must drop these labels itself. A scrape object's own relabel
|
||||
# rules run after these: they may drop the labels but must never set them.
|
||||
# Not covered: VMProbe (operator v0.68.4 takes only auth from a class) and
|
||||
# raw scrape configs. The placeholder is removed by the remote-write
|
||||
# relabeling below.
|
||||
scrapeClasses:
|
||||
- name: application-identity
|
||||
default: true
|
||||
relabelConfigs:
|
||||
{{- range list "group" "kind" "name" }}
|
||||
- targetLabel: apps_cozystack_io_application_{{ . }}
|
||||
replacement: "-"
|
||||
- sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_{{ . }}]
|
||||
regex: (.+)
|
||||
targetLabel: apps_cozystack_io_application_{{ . }}
|
||||
{{- end }}
|
||||
metricRelabelConfigs:
|
||||
- action: labeldrop
|
||||
regex: (exported_)+apps_cozystack_io_application_(group|kind|name)
|
||||
# Not in metricRelabelConfigs: vmagent adds up and scrape_* after metric
|
||||
# relabeling, so the placeholder would stay on those series.
|
||||
inlineRelabelConfig:
|
||||
{{- range list "group" "kind" "name" }}
|
||||
- sourceLabels: [apps_cozystack_io_application_{{ . }}]
|
||||
regex: "-"
|
||||
targetLabel: apps_cozystack_io_application_{{ . }}
|
||||
replacement: ""
|
||||
{{- end }}
|
||||
additionalScrapeConfigs:
|
||||
name: additional-scrape-configs
|
||||
key: prometheus-additional.yaml
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
suite: cluster-wide vmagent stamps application identity on exporter series
|
||||
templates:
|
||||
- templates/vmagent.yaml
|
||||
release:
|
||||
name: monitoring-agents
|
||||
namespace: cozy-monitoring
|
||||
|
||||
# The tenant-resource-metrics proxy scopes a query to one application by
|
||||
# apps_cozystack_io_application_{group,kind,name}. The labels are only a
|
||||
# boundary if the scraped pod cannot choose them: the class sets all three
|
||||
# on every target it covers (a placeholder when the pod has no lineage), so with
|
||||
# honorLabels false an exporter's own copy is renamed to exported_* and
|
||||
# dropped. The same rules live in the tenant vmagent of the monitoring
|
||||
# package and are pinned there by the same assertions.
|
||||
tests:
|
||||
- it: every scrape gets the application identity class unless it names another
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.scrapeClasses[0].name
|
||||
value: application-identity
|
||||
- equal:
|
||||
path: spec.scrapeClasses[0].default
|
||||
value: true
|
||||
- lengthEqual:
|
||||
path: spec.scrapeClasses
|
||||
count: 1
|
||||
|
||||
- it: the identity comes from the scraped pod's lineage labels, with a placeholder when the pod has none
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.scrapeClasses[0].relabelConfigs
|
||||
value:
|
||||
- targetLabel: apps_cozystack_io_application_group
|
||||
replacement: "-"
|
||||
- sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_group]
|
||||
regex: (.+)
|
||||
targetLabel: apps_cozystack_io_application_group
|
||||
- targetLabel: apps_cozystack_io_application_kind
|
||||
replacement: "-"
|
||||
- sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_kind]
|
||||
regex: (.+)
|
||||
targetLabel: apps_cozystack_io_application_kind
|
||||
- targetLabel: apps_cozystack_io_application_name
|
||||
replacement: "-"
|
||||
- sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_name]
|
||||
regex: (.+)
|
||||
targetLabel: apps_cozystack_io_application_name
|
||||
|
||||
- it: an exporter-supplied identity is dropped
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.scrapeClasses[0].metricRelabelConfigs
|
||||
value:
|
||||
- action: labeldrop
|
||||
regex: (exported_)+apps_cozystack_io_application_(group|kind|name)
|
||||
|
||||
# vmagent adds up and scrape_* after metric relabeling (VictoriaMetrics
|
||||
# lib/promscrape addAutoMetrics calls addRows with needRelabel false), so a
|
||||
# placeholder removed there would stay on those series of every target
|
||||
# without lineage. The remote-write relabeling sees every series.
|
||||
- it: the placeholder is removed from every series, including up and scrape_*
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.inlineRelabelConfig
|
||||
value:
|
||||
- sourceLabels: [apps_cozystack_io_application_group]
|
||||
regex: "-"
|
||||
targetLabel: apps_cozystack_io_application_group
|
||||
replacement: ""
|
||||
- sourceLabels: [apps_cozystack_io_application_kind]
|
||||
regex: "-"
|
||||
targetLabel: apps_cozystack_io_application_kind
|
||||
replacement: ""
|
||||
- sourceLabels: [apps_cozystack_io_application_name]
|
||||
regex: "-"
|
||||
targetLabel: apps_cozystack_io_application_name
|
||||
replacement: ""
|
||||
@@ -24,6 +24,39 @@ spec:
|
||||
|
||||
scrapeInterval: 30s
|
||||
selectAllByDefault: false
|
||||
# The class sets all three identity labels on every target it covers, "-"
|
||||
# when the pod has no lineage ("-" is not a valid Kubernetes label value,
|
||||
# so it never collides with a real one). Under honorLabels false every
|
||||
# exporter-supplied copy then collides, is renamed to exported_* and
|
||||
# dropped. Under honorLabels true the exporter's copy wins, so such a
|
||||
# scrape must drop these labels itself. A scrape object's own relabel
|
||||
# rules run after these: they may drop the labels but must never set them.
|
||||
# Not covered: VMProbe (operator v0.68.4 takes only auth from a class) and
|
||||
# raw scrape configs. The placeholder is removed by the remote-write
|
||||
# relabeling below.
|
||||
scrapeClasses:
|
||||
- name: application-identity
|
||||
default: true
|
||||
relabelConfigs:
|
||||
{{- range list "group" "kind" "name" }}
|
||||
- targetLabel: apps_cozystack_io_application_{{ . }}
|
||||
replacement: "-"
|
||||
- sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_{{ . }}]
|
||||
regex: (.+)
|
||||
targetLabel: apps_cozystack_io_application_{{ . }}
|
||||
{{- end }}
|
||||
metricRelabelConfigs:
|
||||
- action: labeldrop
|
||||
regex: (exported_)+apps_cozystack_io_application_(group|kind|name)
|
||||
# Not in metricRelabelConfigs: vmagent adds up and scrape_* after metric
|
||||
# relabeling, so the placeholder would stay on those series.
|
||||
inlineRelabelConfig:
|
||||
{{- range list "group" "kind" "name" }}
|
||||
- sourceLabels: [apps_cozystack_io_application_{{ . }}]
|
||||
regex: "-"
|
||||
targetLabel: apps_cozystack_io_application_{{ . }}
|
||||
replacement: ""
|
||||
{{- end }}
|
||||
{{- with .Values.vmagent.inlineScrapeConfig }}
|
||||
inlineScrapeConfig: |
|
||||
{{- . | nindent 4 }}
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
suite: tenant vmagent stamps application identity on exporter series
|
||||
templates:
|
||||
- templates/vm/vmagent.yaml
|
||||
release:
|
||||
name: monitoring
|
||||
namespace: tenant-foo
|
||||
|
||||
# Same rules as the cluster-wide vmagent in monitoring-agents, pinned there
|
||||
# by the same assertions; see that suite for why the placeholder and the
|
||||
# drop exist.
|
||||
tests:
|
||||
- it: every scrape gets the application identity class unless it names another
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.scrapeClasses[0].name
|
||||
value: application-identity
|
||||
- equal:
|
||||
path: spec.scrapeClasses[0].default
|
||||
value: true
|
||||
- lengthEqual:
|
||||
path: spec.scrapeClasses
|
||||
count: 1
|
||||
|
||||
- it: the identity comes from the scraped pod's lineage labels, with a placeholder when the pod has none
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.scrapeClasses[0].relabelConfigs
|
||||
value:
|
||||
- targetLabel: apps_cozystack_io_application_group
|
||||
replacement: "-"
|
||||
- sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_group]
|
||||
regex: (.+)
|
||||
targetLabel: apps_cozystack_io_application_group
|
||||
- targetLabel: apps_cozystack_io_application_kind
|
||||
replacement: "-"
|
||||
- sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_kind]
|
||||
regex: (.+)
|
||||
targetLabel: apps_cozystack_io_application_kind
|
||||
- targetLabel: apps_cozystack_io_application_name
|
||||
replacement: "-"
|
||||
- sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_name]
|
||||
regex: (.+)
|
||||
targetLabel: apps_cozystack_io_application_name
|
||||
|
||||
- it: an exporter-supplied identity is dropped
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.scrapeClasses[0].metricRelabelConfigs
|
||||
value:
|
||||
- action: labeldrop
|
||||
regex: (exported_)+apps_cozystack_io_application_(group|kind|name)
|
||||
|
||||
# vmagent adds up and scrape_* after metric relabeling (VictoriaMetrics
|
||||
# lib/promscrape addAutoMetrics calls addRows with needRelabel false), so a
|
||||
# placeholder removed there would stay on those series of every target
|
||||
# without lineage. The remote-write relabeling sees every series.
|
||||
- it: the placeholder is removed from every series, including up and scrape_*
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.inlineRelabelConfig
|
||||
value:
|
||||
- sourceLabels: [apps_cozystack_io_application_group]
|
||||
regex: "-"
|
||||
targetLabel: apps_cozystack_io_application_group
|
||||
replacement: ""
|
||||
- sourceLabels: [apps_cozystack_io_application_kind]
|
||||
regex: "-"
|
||||
targetLabel: apps_cozystack_io_application_kind
|
||||
replacement: ""
|
||||
- sourceLabels: [apps_cozystack_io_application_name]
|
||||
regex: "-"
|
||||
targetLabel: apps_cozystack_io_application_name
|
||||
replacement: ""
|
||||
Reference in new issue
Block a user