mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 20:23:39 +09:00
fix(piraeus-operator): keep the LINSTOR error-report alerts quiet when the controller pod moves (#4963)
## What this PR does `linstorSatelliteErrorRate` fires for every node that has any old LINSTOR error report whenever the LINSTOR controller pod moves to another node, and it keeps firing for 15 minutes. `linstor_error_reports_count` is exported by the controller, so its series carry the controller pod's `instance` and `controller_node`, and a moved pod starts new series that hold the existing counts. VictoriaMetrics' [`increase()`](https://docs.victoriametrics.com/victoriametrics/metricsql/#increase) starts from the last sample before the window, and when a series has none, [`rollupDelta`](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/v1.136.0/app/vmselect/promql/rollup.go#L1859-L1901) counts it from zero whenever its first value is below ten times the next change plus one. That is the case the docs describe as `increase` ignoring "the first value in a series if it is too big" under [`increase_pure`](https://docs.victoriametrics.com/victoriametrics/metricsql/#increase_pure), and report counts are never that big, so every new series counts all of its reports as new. The `up` guard from #2265 matches `on(instance, job)`, and the new controller is up, so it lets them through. Seen on a v1.6.5 cluster with VictoriaMetrics v1.136.0. During the upgrade the controller pod moved to another node, and the alert fired for all three nodes with values 1, 1 and 2 for 15 minutes, while `linstor error-reports list --since 1h` was empty and the four reports that exist were weeks old. The day before, it fired the same way for about 15 minutes when only the `cluster` external label of vmagent changed. A satellite coming back to the metric does the same. In LINSTOR v1.33.3 the controller asks every satellite for its reports on each scrape and leaves out any satellite that is not connected ([`prepareErrRequestApi`](https://github.com/LINBIT/linstor-server/blob/v1.33.3/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlErrorListApiCallHandler.java#L240-L259)), so after a node reboot or any other disconnect longer than five minutes the satellite's old count is counted again. Both rules now take the count per `hostname`, hold the last value for an hour with `last_over_time`, and run `increase()` over that as a one-minute [subquery](https://docs.victoriametrics.com/victoriametrics/metricsql/#subqueries), whose inner points VictoriaMetrics aligns to that step. Keyed on `hostname`, a moved controller or a changed label continues the same series, and the hold bridges a disconnect or a slow controller start of up to an hour. The `up` guard matches `on()`, since the aggregated result has no `instance` any more. A new report still fires with the same value and stays for 15 minutes, at most a minute later than before because of the one-minute step. The alerts now carry `hostname` and `severity`, and no longer the controller's `instance`, `job` and `controller_node`, the `module`, or the external labels of the scrape. `linstorControllerErrorRate` gets the same form. Its `hostname` is the controller pod name and the piraeus controller keeps its reports in an [`emptyDir`](https://github.com/piraeusdatastore/piraeus-operator/blob/v2.10.2/pkg/resources/cluster/controller/controller-deployment.yaml#L111-L112), so a new pod starts with no reports and a move alone does not trip it, but a label change does whenever the controller has reports. A gap longer than the hour is not covered. After it the series starts over, and as before the `up` guard holds the alert back only if the controller target recorded failed scrapes during the gap. I checked it against the stored metrics of the cluster above with read-only queries to vmselect. Over the upgrade the current expression returns the three nodes from 15:45:30 to 16:00:00 UTC at a 30-second step, the same interval as vmalert's `ALERTS` series, and over the label change from 20:27:30 to 20:42:00. The new expression returns nothing over both windows, and without `> 0` it returns each of the three nodes with 0 at every step, 88 and 121 points. The same storage has six gaps of 8 to 113 minutes over one night with no failed scrape recorded in them. Replaying vmalert's instant query of the `increase()` part every 30 seconds in the 154 evaluations after those gaps, the current expression returns at least one node in 99, the `hostname` key without the hold in 84, and the new expression in 30, all of them after the 113-minute gap. Retention there does not reach back to a real new report, so the firing side is covered by the tests. `hack/linstor-error-report-alerts.bats` replays both rules through `vmalert-tool` v1.136.0, pinned by digest, which is the VictoriaMetrics release victoria-metrics-operator v0.68.4 [deploys by default](https://github.com/VictoriaMetrics/operator/blob/v0.68.4/internal/config/config.go#L38), with 30-second samples and 15-second evaluations like the platform's vmagent and vmalert. The suite needs docker, like the migration suites, and only the image pull touches the network. `hack/testdata/linstor-error-report-alerts/unittest.yaml` has 20 cases in five groups, which cover a controller move followed by a new report, a changed external label for both rules, one satellite away for 40 minutes and one that comes back with a new report, a new report and a first report on a node for both rules, and a controller that cannot be scraped for 70 minutes. Against the rule on main 13 cases fail, 9 of them because old reports fire again after the move, the label change or the disconnect, and the other 4 on the label set alone, since the current alerts carry the controller's labels. Each of seven mutations of the new expressions turns at least one case red, namely no hold, a 15-minute hold, the guard on `instance`, no guard, the key on `hostname` and `instance`, the key on `hostname` and `cluster`, and the controller rule as on main. With Bats 1.14.0 at the commit CI pins, the new suite with `alert-severity-contract`, `bats-strict-setup`, `bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage` and `image-pin-consistency` passes 74 of 74, and `helm template` renders the same 15 rules as main. I have not run the new rules in vmalert on a live cluster. The commit cherry-picks onto release-1.7 as is, and the suite passes there. On release-1.6 it applies without conflicts, but that branch runs `hack/*.bats` through `hack/cozytest.sh` and has no `hack/test_helper.bash`, so the suite there has to drop its `load test_helper` line. ### Screenshots Not a UI change. ### Downstream repositories A rule file under `packages/system/piraeus-operator/alerts/`, a new Bats suite under `hack/` and its test data. No values, schema, README, image, make target or `ApplicationDefinition` change, and the website does not mention these rules or the metric, so no trigger map entry applies. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(piraeus-operator): the LINSTOR error-report alerts no longer fire on old reports when the LINSTOR controller pod moves to another node, when an external label of the scrape changes, or when a satellite reconnects within an hour. They now count reports per node, and their alerts carry the node's hostname instead of the controller pod's instance, job and controller_node labels. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved LINSTOR controller and satellite error alerts to account for recent error reports over a 15-minute window, even when metrics are intermittent or scrape labels change. * Updated alert matching to better handle controller moves between nodes and satellites that disappear from metrics while disconnected. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
3 files changed
+233
-3
No files matched your search
@@ -0,0 +1,32 @@
|
||||
#!/usr/bin/env bats
|
||||
# Replays the LINSTOR error-report alerts from
|
||||
# packages/system/piraeus-operator/alerts/piraeus-datastore.yaml through
|
||||
# vmalert-tool, pinned to the VictoriaMetrics release that the
|
||||
# victoria-metrics-operator deploys by default. Its increase() counts the first
|
||||
# samples of a series as new when they are small, and the report counts are
|
||||
# small, so the cases pin that a series which starts over after a controller
|
||||
# move, a label change or a disconnect does not report old error reports again.
|
||||
# Only pulling the image touches the network.
|
||||
|
||||
load test_helper
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
|
||||
VMALERT_TOOL_IMAGE="victoriametrics/vmalert-tool:v1.136.0@sha256:417346d7b6fdb66d8e3984887c2ad114535ec7d983eeef53381b56c776f333f1"
|
||||
|
||||
@test "LINSTOR error-report alerts pass their vmalert-tool cases" {
|
||||
docker info >/dev/null 2>&1 || {
|
||||
echo "docker is required to run the cases in $VMALERT_TOOL_IMAGE" >&2
|
||||
return 1
|
||||
}
|
||||
work=$(mktemp -d)
|
||||
yq '.spec' "$REPO_ROOT/packages/system/piraeus-operator/alerts/piraeus-datastore.yaml" > "$work/rules.yaml"
|
||||
cp "$REPO_ROOT/hack/testdata/linstor-error-report-alerts/unittest.yaml" "$work/unittest.yaml"
|
||||
rc=0
|
||||
docker run --rm --network none -v "$work:/w:ro" "$VMALERT_TOOL_IMAGE" \
|
||||
unittest --files /w/unittest.yaml > "$work/out" 2>&1 || rc=$?
|
||||
cat "$work/out"
|
||||
passed=$(grep -cx 'SUCCESS' "$work/out" || true)
|
||||
rm -rf "$work"
|
||||
[ "$rc" -eq 0 ]
|
||||
[ "$passed" -eq 1 ]
|
||||
}
|
||||
@@ -0,0 +1,198 @@
|
||||
rule_files:
|
||||
- rules.yaml
|
||||
|
||||
evaluation_interval: 15s
|
||||
|
||||
tests:
|
||||
- name: controller pod moves to another node
|
||||
interval: 30s
|
||||
input_series:
|
||||
- series: 'up{job="linstor-controller", instance="10.0.0.1:3370", controller_node="node-a"}'
|
||||
values: '1x38 0 stale'
|
||||
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", controller_node="node-a", hostname="node-a", module="SATELLITE"}'
|
||||
values: '1x38 stale'
|
||||
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", controller_node="node-a", hostname="node-b", module="SATELLITE"}'
|
||||
values: '2x38 stale'
|
||||
- series: 'up{job="linstor-controller", instance="10.0.0.2:3370", controller_node="node-b"}'
|
||||
values: '_x41 1x79'
|
||||
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.2:3370", controller_node="node-b", hostname="node-a", module="SATELLITE"}'
|
||||
values: '_x41 1x58 2x20'
|
||||
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.2:3370", controller_node="node-b", hostname="node-b", module="SATELLITE"}'
|
||||
values: '_x41 2x79'
|
||||
alert_rule_test:
|
||||
- eval_time: 21m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorSatelliteErrorRate
|
||||
- eval_time: 28m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorSatelliteErrorRate
|
||||
- eval_time: 35m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorSatelliteErrorRate
|
||||
- eval_time: 45m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorSatelliteErrorRate
|
||||
- eval_time: 51m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorSatelliteErrorRate
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
hostname: node-a
|
||||
severity: warning
|
||||
exp_annotations:
|
||||
description: |
|
||||
LINSTOR Satellite "node-a" reports 1 errors in the last 15 minutes.
|
||||
Use "linstor error-reports list --nodes node-a" to inspect the reports.
|
||||
|
||||
- name: an external label of the scrape changes
|
||||
interval: 30s
|
||||
input_series:
|
||||
- series: 'up{job="linstor-controller", instance="10.0.0.1:3370", cluster="cozystack"}'
|
||||
values: '1x39 stale'
|
||||
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", cluster="cozystack", hostname="node-a", module="SATELLITE"}'
|
||||
values: '1x39 stale'
|
||||
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", cluster="cozystack", hostname="linstor-controller-abc", module="CONTROLLER"}'
|
||||
values: '1x39 stale'
|
||||
- series: 'up{job="linstor-controller", instance="10.0.0.1:3370", cluster="cluster-a"}'
|
||||
values: '_x41 1x79'
|
||||
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", cluster="cluster-a", hostname="node-a", module="SATELLITE"}'
|
||||
values: '_x41 1x79'
|
||||
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", cluster="cluster-a", hostname="linstor-controller-abc", module="CONTROLLER"}'
|
||||
values: '_x41 1x79'
|
||||
alert_rule_test:
|
||||
- eval_time: 21m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorSatelliteErrorRate
|
||||
- eval_time: 30m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorSatelliteErrorRate
|
||||
- eval_time: 21m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorControllerErrorRate
|
||||
- eval_time: 30m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorControllerErrorRate
|
||||
|
||||
- name: a satellite drops out of the metric while it is disconnected
|
||||
interval: 30s
|
||||
input_series:
|
||||
- series: 'up{job="linstor-controller", instance="10.0.0.1:3370"}'
|
||||
values: '1x220'
|
||||
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-a", module="SATELLITE"}'
|
||||
values: '1x38 stale _x80 1x100'
|
||||
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-b", module="SATELLITE"}'
|
||||
values: '2x220'
|
||||
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-c", module="SATELLITE"}'
|
||||
values: '1x38 stale _x80 2x100'
|
||||
alert_rule_test:
|
||||
- eval_time: 61m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorSatelliteErrorRate
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
hostname: node-c
|
||||
severity: warning
|
||||
exp_annotations:
|
||||
description: |
|
||||
LINSTOR Satellite "node-c" reports 1 errors in the last 15 minutes.
|
||||
Use "linstor error-reports list --nodes node-c" to inspect the reports.
|
||||
- eval_time: 70m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorSatelliteErrorRate
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
hostname: node-c
|
||||
severity: warning
|
||||
exp_annotations:
|
||||
description: |
|
||||
LINSTOR Satellite "node-c" reports 1 errors in the last 15 minutes.
|
||||
Use "linstor error-reports list --nodes node-c" to inspect the reports.
|
||||
- eval_time: 76m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorSatelliteErrorRate
|
||||
|
||||
- name: a new report fires for the 15 minute window
|
||||
interval: 30s
|
||||
input_series:
|
||||
- series: 'up{job="linstor-controller", instance="10.0.0.1:3370"}'
|
||||
values: '1x120'
|
||||
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-a", module="SATELLITE"}'
|
||||
values: '1x39 2x80'
|
||||
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-b", module="SATELLITE"}'
|
||||
values: '2x120'
|
||||
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-c", module="SATELLITE"}'
|
||||
values: '0x39 1x80'
|
||||
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="linstor-controller-abc", module="CONTROLLER"}'
|
||||
values: '0x39 1x80'
|
||||
alert_rule_test:
|
||||
- eval_time: 21m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorSatelliteErrorRate
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
hostname: node-a
|
||||
severity: warning
|
||||
exp_annotations:
|
||||
description: |
|
||||
LINSTOR Satellite "node-a" reports 1 errors in the last 15 minutes.
|
||||
Use "linstor error-reports list --nodes node-a" to inspect the reports.
|
||||
- exp_labels:
|
||||
hostname: node-c
|
||||
severity: warning
|
||||
exp_annotations:
|
||||
description: |
|
||||
LINSTOR Satellite "node-c" reports 1 errors in the last 15 minutes.
|
||||
Use "linstor error-reports list --nodes node-c" to inspect the reports.
|
||||
- eval_time: 34m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorSatelliteErrorRate
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
hostname: node-a
|
||||
severity: warning
|
||||
exp_annotations:
|
||||
description: |
|
||||
LINSTOR Satellite "node-a" reports 1 errors in the last 15 minutes.
|
||||
Use "linstor error-reports list --nodes node-a" to inspect the reports.
|
||||
- exp_labels:
|
||||
hostname: node-c
|
||||
severity: warning
|
||||
exp_annotations:
|
||||
description: |
|
||||
LINSTOR Satellite "node-c" reports 1 errors in the last 15 minutes.
|
||||
Use "linstor error-reports list --nodes node-c" to inspect the reports.
|
||||
- eval_time: 36m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorSatelliteErrorRate
|
||||
- eval_time: 21m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorControllerErrorRate
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
hostname: linstor-controller-abc
|
||||
severity: warning
|
||||
exp_annotations:
|
||||
description: |
|
||||
LINSTOR Controller reports 1 errors in the last 15 minutes.
|
||||
Use "linstor error-reports list --since 15minutes" to see them.
|
||||
- eval_time: 36m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorControllerErrorRate
|
||||
|
||||
- name: a failed controller scrape still holds the alert back
|
||||
interval: 30s
|
||||
input_series:
|
||||
- series: 'up{job="linstor-controller", instance="10.0.0.1:3370"}'
|
||||
values: '1x19 0x140 1x60'
|
||||
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-a", module="SATELLITE"}'
|
||||
values: '1x19 stale _x140 1x60'
|
||||
alert_rule_test:
|
||||
- eval_time: 85m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorSatelliteErrorRate
|
||||
- eval_time: 93m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorSatelliteErrorRate
|
||||
- eval_time: 100m
|
||||
groupname: linstor.rules
|
||||
alertname: linstorSatelliteErrorRate
|
||||
@@ -29,8 +29,8 @@ spec:
|
||||
LINSTOR Satellite "{{ $labels.hostname }}" reports {{ $value }} errors in the last 15 minutes.
|
||||
Use "linstor error-reports list --nodes {{ $labels.hostname }}" to inspect the reports.
|
||||
expr: |
|
||||
increase(linstor_error_reports_count{module="SATELLITE"}[15m]) > 0
|
||||
and on(instance, job)
|
||||
increase((max by (hostname) (last_over_time(linstor_error_reports_count{module="SATELLITE"}[1h])))[15m:1m]) > 0
|
||||
and on()
|
||||
min_over_time(up{job="linstor-controller"}[15m]) == 1
|
||||
labels:
|
||||
severity: warning
|
||||
@@ -39,7 +39,7 @@ spec:
|
||||
description: |
|
||||
LINSTOR Controller reports {{ $value }} errors in the last 15 minutes.
|
||||
Use "linstor error-reports list --since 15minutes" to see them.
|
||||
expr: increase(linstor_error_reports_count{module="CONTROLLER"}[15m]) > 0
|
||||
expr: increase((max by (hostname) (last_over_time(linstor_error_reports_count{module="CONTROLLER"}[1h])))[15m:1m]) > 0
|
||||
labels:
|
||||
severity: warning
|
||||
- alert: linstorSatelliteNotOnline
|
||||
|
||||
Reference in new issue
Block a user