fix(piraeus-operator): keep the LINSTOR error-report alerts quiet when the controller pod moves (#4963)

## What this PR does

`linstorSatelliteErrorRate` fires for every node that has any old
LINSTOR error report whenever the LINSTOR controller pod moves to
another node, and it keeps firing for 15 minutes.
`linstor_error_reports_count` is exported by the controller, so its
series carry the controller pod's `instance` and `controller_node`, and
a moved pod starts new series that hold the existing counts.
VictoriaMetrics'
[`increase()`](https://docs.victoriametrics.com/victoriametrics/metricsql/#increase)
starts from the last sample before the window, and when a series has
none,
[`rollupDelta`](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/v1.136.0/app/vmselect/promql/rollup.go#L1859-L1901)
counts it from zero whenever its first value is below ten times the next
change plus one. That is the case the docs describe as `increase`
ignoring "the first value in a series if it is too big" under
[`increase_pure`](https://docs.victoriametrics.com/victoriametrics/metricsql/#increase_pure),
and report counts are never that big, so every new series counts all of
its reports as new. The `up` guard from #2265 matches `on(instance,
job)`, and the new controller is up, so it lets them through.

Seen on a v1.6.5 cluster with VictoriaMetrics v1.136.0. During the
upgrade the controller pod moved to another node, and the alert fired
for all three nodes with values 1, 1 and 2 for 15 minutes, while
`linstor error-reports list --since 1h` was empty and the four reports
that exist were weeks old. The day before, it fired the same way for
about 15 minutes when only the `cluster` external label of vmagent
changed.

A satellite coming back to the metric does the same. In LINSTOR v1.33.3
the controller asks every satellite for its reports on each scrape and
leaves out any satellite that is not connected
([`prepareErrRequestApi`](https://github.com/LINBIT/linstor-server/blob/v1.33.3/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlErrorListApiCallHandler.java#L240-L259)),
so after a node reboot or any other disconnect longer than five minutes
the satellite's old count is counted again.

Both rules now take the count per `hostname`, hold the last value for an
hour with `last_over_time`, and run `increase()` over that as a
one-minute
[subquery](https://docs.victoriametrics.com/victoriametrics/metricsql/#subqueries),
whose inner points VictoriaMetrics aligns to that step. Keyed on
`hostname`, a moved controller or a changed label continues the same
series, and the hold bridges a disconnect or a slow controller start of
up to an hour. The `up` guard matches `on()`, since the aggregated
result has no `instance` any more. A new report still fires with the
same value and stays for 15 minutes, at most a minute later than before
because of the one-minute step. The alerts now carry `hostname` and
`severity`, and no longer the controller's `instance`, `job` and
`controller_node`, the `module`, or the external labels of the scrape.

`linstorControllerErrorRate` gets the same form. Its `hostname` is the
controller pod name and the piraeus controller keeps its reports in an
[`emptyDir`](https://github.com/piraeusdatastore/piraeus-operator/blob/v2.10.2/pkg/resources/cluster/controller/controller-deployment.yaml#L111-L112),
so a new pod starts with no reports and a move alone does not trip it,
but a label change does whenever the controller has reports.

A gap longer than the hour is not covered. After it the series starts
over, and as before the `up` guard holds the alert back only if the
controller target recorded failed scrapes during the gap.

I checked it against the stored metrics of the cluster above with
read-only queries to vmselect. Over the upgrade the current expression
returns the three nodes from 15:45:30 to 16:00:00 UTC at a 30-second
step, the same interval as vmalert's `ALERTS` series, and over the label
change from 20:27:30 to 20:42:00. The new expression returns nothing
over both windows, and without `> 0` it returns each of the three nodes
with 0 at every step, 88 and 121 points. The same storage has six gaps
of 8 to 113 minutes over one night with no failed scrape recorded in
them. Replaying vmalert's instant query of the `increase()` part every
30 seconds in the 154 evaluations after those gaps, the current
expression returns at least one node in 99, the `hostname` key without
the hold in 84, and the new expression in 30, all of them after the
113-minute gap. Retention there does not reach back to a real new
report, so the firing side is covered by the tests.

`hack/linstor-error-report-alerts.bats` replays both rules through
`vmalert-tool` v1.136.0, pinned by digest, which is the VictoriaMetrics
release victoria-metrics-operator v0.68.4 [deploys by
default](https://github.com/VictoriaMetrics/operator/blob/v0.68.4/internal/config/config.go#L38),
with 30-second samples and 15-second evaluations like the platform's
vmagent and vmalert. The suite needs docker, like the migration suites,
and only the image pull touches the network.
`hack/testdata/linstor-error-report-alerts/unittest.yaml` has 20 cases
in five groups, which cover a controller move followed by a new report,
a changed external label for both rules, one satellite away for 40
minutes and one that comes back with a new report, a new report and a
first report on a node for both rules, and a controller that cannot be
scraped for 70 minutes. Against the rule on main 13 cases fail, 9 of
them because old reports fire again after the move, the label change or
the disconnect, and the other 4 on the label set alone, since the
current alerts carry the controller's labels. Each of seven mutations of
the new expressions turns at least one case red, namely no hold, a
15-minute hold, the guard on `instance`, no guard, the key on `hostname`
and `instance`, the key on `hostname` and `cluster`, and the controller
rule as on main. With Bats 1.14.0 at the commit CI pins, the new suite
with `alert-severity-contract`, `bats-strict-setup`,
`bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage`
and `image-pin-consistency` passes 74 of 74, and `helm template` renders
the same 15 rules as main. I have not run the new rules in vmalert on a
live cluster.

The commit cherry-picks onto release-1.7 as is, and the suite passes
there. On release-1.6 it applies without conflicts, but that branch runs
`hack/*.bats` through `hack/cozytest.sh` and has no
`hack/test_helper.bash`, so the suite there has to drop its `load
test_helper` line.

### Screenshots

Not a UI change.

### Downstream repositories

A rule file under `packages/system/piraeus-operator/alerts/`, a new Bats
suite under `hack/` and its test data. No values, schema, README, image,
make target or `ApplicationDefinition` change, and the website does not
mention these rules or the metric, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(piraeus-operator): the LINSTOR error-report alerts no longer fire on old reports when the LINSTOR controller pod moves to another node, when an external label of the scrape changes, or when a satellite reconnects within an hour. They now count reports per node, and their alerts carry the node's hostname instead of the controller pod's instance, job and controller_node labels.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved LINSTOR controller and satellite error alerts to account for
recent error reports over a 15-minute window, even when metrics are
intermittent or scrape labels change.
* Updated alert matching to better handle controller moves between nodes
and satellites that disappear from metrics while disconnected.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
Aleksei Sviridkin authored and GitHub committed 2026-11-13 18:18:47 +03:00
commit 51282a1243
3 files changed
+233 -3

No files matched your search

+32
View File
@@ -0,0 +1,32 @@
#!/usr/bin/env bats
# Replays the LINSTOR error-report alerts from
# packages/system/piraeus-operator/alerts/piraeus-datastore.yaml through
# vmalert-tool, pinned to the VictoriaMetrics release that the
# victoria-metrics-operator deploys by default. Its increase() counts the first
# samples of a series as new when they are small, and the report counts are
# small, so the cases pin that a series which starts over after a controller
# move, a label change or a disconnect does not report old error reports again.
# Only pulling the image touches the network.
load test_helper
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
VMALERT_TOOL_IMAGE="victoriametrics/vmalert-tool:v1.136.0@sha256:417346d7b6fdb66d8e3984887c2ad114535ec7d983eeef53381b56c776f333f1"
@test "LINSTOR error-report alerts pass their vmalert-tool cases" {
docker info >/dev/null 2>&1 || {
echo "docker is required to run the cases in $VMALERT_TOOL_IMAGE" >&2
return 1
}
work=$(mktemp -d)
yq '.spec' "$REPO_ROOT/packages/system/piraeus-operator/alerts/piraeus-datastore.yaml" > "$work/rules.yaml"
cp "$REPO_ROOT/hack/testdata/linstor-error-report-alerts/unittest.yaml" "$work/unittest.yaml"
rc=0
docker run --rm --network none -v "$work:/w:ro" "$VMALERT_TOOL_IMAGE" \
unittest --files /w/unittest.yaml > "$work/out" 2>&1 || rc=$?
cat "$work/out"
passed=$(grep -cx 'SUCCESS' "$work/out" || true)
rm -rf "$work"
[ "$rc" -eq 0 ]
[ "$passed" -eq 1 ]
}
+198
View File
@@ -0,0 +1,198 @@
rule_files:
- rules.yaml
evaluation_interval: 15s
tests:
- name: controller pod moves to another node
interval: 30s
input_series:
- series: 'up{job="linstor-controller", instance="10.0.0.1:3370", controller_node="node-a"}'
values: '1x38 0 stale'
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", controller_node="node-a", hostname="node-a", module="SATELLITE"}'
values: '1x38 stale'
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", controller_node="node-a", hostname="node-b", module="SATELLITE"}'
values: '2x38 stale'
- series: 'up{job="linstor-controller", instance="10.0.0.2:3370", controller_node="node-b"}'
values: '_x41 1x79'
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.2:3370", controller_node="node-b", hostname="node-a", module="SATELLITE"}'
values: '_x41 1x58 2x20'
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.2:3370", controller_node="node-b", hostname="node-b", module="SATELLITE"}'
values: '_x41 2x79'
alert_rule_test:
- eval_time: 21m
groupname: linstor.rules
alertname: linstorSatelliteErrorRate
- eval_time: 28m
groupname: linstor.rules
alertname: linstorSatelliteErrorRate
- eval_time: 35m
groupname: linstor.rules
alertname: linstorSatelliteErrorRate
- eval_time: 45m
groupname: linstor.rules
alertname: linstorSatelliteErrorRate
- eval_time: 51m
groupname: linstor.rules
alertname: linstorSatelliteErrorRate
exp_alerts:
- exp_labels:
hostname: node-a
severity: warning
exp_annotations:
description: |
LINSTOR Satellite "node-a" reports 1 errors in the last 15 minutes.
Use "linstor error-reports list --nodes node-a" to inspect the reports.
- name: an external label of the scrape changes
interval: 30s
input_series:
- series: 'up{job="linstor-controller", instance="10.0.0.1:3370", cluster="cozystack"}'
values: '1x39 stale'
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", cluster="cozystack", hostname="node-a", module="SATELLITE"}'
values: '1x39 stale'
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", cluster="cozystack", hostname="linstor-controller-abc", module="CONTROLLER"}'
values: '1x39 stale'
- series: 'up{job="linstor-controller", instance="10.0.0.1:3370", cluster="cluster-a"}'
values: '_x41 1x79'
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", cluster="cluster-a", hostname="node-a", module="SATELLITE"}'
values: '_x41 1x79'
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", cluster="cluster-a", hostname="linstor-controller-abc", module="CONTROLLER"}'
values: '_x41 1x79'
alert_rule_test:
- eval_time: 21m
groupname: linstor.rules
alertname: linstorSatelliteErrorRate
- eval_time: 30m
groupname: linstor.rules
alertname: linstorSatelliteErrorRate
- eval_time: 21m
groupname: linstor.rules
alertname: linstorControllerErrorRate
- eval_time: 30m
groupname: linstor.rules
alertname: linstorControllerErrorRate
- name: a satellite drops out of the metric while it is disconnected
interval: 30s
input_series:
- series: 'up{job="linstor-controller", instance="10.0.0.1:3370"}'
values: '1x220'
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-a", module="SATELLITE"}'
values: '1x38 stale _x80 1x100'
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-b", module="SATELLITE"}'
values: '2x220'
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-c", module="SATELLITE"}'
values: '1x38 stale _x80 2x100'
alert_rule_test:
- eval_time: 61m
groupname: linstor.rules
alertname: linstorSatelliteErrorRate
exp_alerts:
- exp_labels:
hostname: node-c
severity: warning
exp_annotations:
description: |
LINSTOR Satellite "node-c" reports 1 errors in the last 15 minutes.
Use "linstor error-reports list --nodes node-c" to inspect the reports.
- eval_time: 70m
groupname: linstor.rules
alertname: linstorSatelliteErrorRate
exp_alerts:
- exp_labels:
hostname: node-c
severity: warning
exp_annotations:
description: |
LINSTOR Satellite "node-c" reports 1 errors in the last 15 minutes.
Use "linstor error-reports list --nodes node-c" to inspect the reports.
- eval_time: 76m
groupname: linstor.rules
alertname: linstorSatelliteErrorRate
- name: a new report fires for the 15 minute window
interval: 30s
input_series:
- series: 'up{job="linstor-controller", instance="10.0.0.1:3370"}'
values: '1x120'
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-a", module="SATELLITE"}'
values: '1x39 2x80'
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-b", module="SATELLITE"}'
values: '2x120'
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-c", module="SATELLITE"}'
values: '0x39 1x80'
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="linstor-controller-abc", module="CONTROLLER"}'
values: '0x39 1x80'
alert_rule_test:
- eval_time: 21m
groupname: linstor.rules
alertname: linstorSatelliteErrorRate
exp_alerts:
- exp_labels:
hostname: node-a
severity: warning
exp_annotations:
description: |
LINSTOR Satellite "node-a" reports 1 errors in the last 15 minutes.
Use "linstor error-reports list --nodes node-a" to inspect the reports.
- exp_labels:
hostname: node-c
severity: warning
exp_annotations:
description: |
LINSTOR Satellite "node-c" reports 1 errors in the last 15 minutes.
Use "linstor error-reports list --nodes node-c" to inspect the reports.
- eval_time: 34m
groupname: linstor.rules
alertname: linstorSatelliteErrorRate
exp_alerts:
- exp_labels:
hostname: node-a
severity: warning
exp_annotations:
description: |
LINSTOR Satellite "node-a" reports 1 errors in the last 15 minutes.
Use "linstor error-reports list --nodes node-a" to inspect the reports.
- exp_labels:
hostname: node-c
severity: warning
exp_annotations:
description: |
LINSTOR Satellite "node-c" reports 1 errors in the last 15 minutes.
Use "linstor error-reports list --nodes node-c" to inspect the reports.
- eval_time: 36m
groupname: linstor.rules
alertname: linstorSatelliteErrorRate
- eval_time: 21m
groupname: linstor.rules
alertname: linstorControllerErrorRate
exp_alerts:
- exp_labels:
hostname: linstor-controller-abc
severity: warning
exp_annotations:
description: |
LINSTOR Controller reports 1 errors in the last 15 minutes.
Use "linstor error-reports list --since 15minutes" to see them.
- eval_time: 36m
groupname: linstor.rules
alertname: linstorControllerErrorRate
- name: a failed controller scrape still holds the alert back
interval: 30s
input_series:
- series: 'up{job="linstor-controller", instance="10.0.0.1:3370"}'
values: '1x19 0x140 1x60'
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-a", module="SATELLITE"}'
values: '1x19 stale _x140 1x60'
alert_rule_test:
- eval_time: 85m
groupname: linstor.rules
alertname: linstorSatelliteErrorRate
- eval_time: 93m
groupname: linstor.rules
alertname: linstorSatelliteErrorRate
- eval_time: 100m
groupname: linstor.rules
alertname: linstorSatelliteErrorRate
@@ -29,8 +29,8 @@ spec:
LINSTOR Satellite "{{ $labels.hostname }}" reports {{ $value }} errors in the last 15 minutes.
Use "linstor error-reports list --nodes {{ $labels.hostname }}" to inspect the reports.
expr: |
increase(linstor_error_reports_count{module="SATELLITE"}[15m]) > 0
and on(instance, job)
increase((max by (hostname) (last_over_time(linstor_error_reports_count{module="SATELLITE"}[1h])))[15m:1m]) > 0
and on()
min_over_time(up{job="linstor-controller"}[15m]) == 1
labels:
severity: warning
@@ -39,7 +39,7 @@ spec:
description: |
LINSTOR Controller reports {{ $value }} errors in the last 15 minutes.
Use "linstor error-reports list --since 15minutes" to see them.
expr: increase(linstor_error_reports_count{module="CONTROLLER"}[15m]) > 0
expr: increase((max by (hostname) (last_over_time(linstor_error_reports_count{module="CONTROLLER"}[1h])))[15m:1m]) > 0
labels:
severity: warning
- alert: linstorSatelliteNotOnline