Aleksei Sviridkin 51282a1243 fix(piraeus-operator): keep the LINSTOR error-report alerts quiet when the controller pod moves (#4963)
## What this PR does

`linstorSatelliteErrorRate` fires for every node that has any old
LINSTOR error report whenever the LINSTOR controller pod moves to
another node, and it keeps firing for 15 minutes.
`linstor_error_reports_count` is exported by the controller, so its
series carry the controller pod's `instance` and `controller_node`, and
a moved pod starts new series that hold the existing counts.
VictoriaMetrics'
[`increase()`](https://docs.victoriametrics.com/victoriametrics/metricsql/#increase)
starts from the last sample before the window, and when a series has
none,
[`rollupDelta`](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/v1.136.0/app/vmselect/promql/rollup.go#L1859-L1901)
counts it from zero whenever its first value is below ten times the next
change plus one. That is the case the docs describe as `increase`
ignoring "the first value in a series if it is too big" under
[`increase_pure`](https://docs.victoriametrics.com/victoriametrics/metricsql/#increase_pure),
and report counts are never that big, so every new series counts all of
its reports as new. The `up` guard from #2265 matches `on(instance,
job)`, and the new controller is up, so it lets them through.

Seen on a v1.6.5 cluster with VictoriaMetrics v1.136.0. During the
upgrade the controller pod moved to another node, and the alert fired
for all three nodes with values 1, 1 and 2 for 15 minutes, while
`linstor error-reports list --since 1h` was empty and the four reports
that exist were weeks old. The day before, it fired the same way for
about 15 minutes when only the `cluster` external label of vmagent
changed.

A satellite coming back to the metric does the same. In LINSTOR v1.33.3
the controller asks every satellite for its reports on each scrape and
leaves out any satellite that is not connected
([`prepareErrRequestApi`](https://github.com/LINBIT/linstor-server/blob/v1.33.3/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlErrorListApiCallHandler.java#L240-L259)),
so after a node reboot or any other disconnect longer than five minutes
the satellite's old count is counted again.

Both rules now take the count per `hostname`, hold the last value for an
hour with `last_over_time`, and run `increase()` over that as a
one-minute
[subquery](https://docs.victoriametrics.com/victoriametrics/metricsql/#subqueries),
whose inner points VictoriaMetrics aligns to that step. Keyed on
`hostname`, a moved controller or a changed label continues the same
series, and the hold bridges a disconnect or a slow controller start of
up to an hour. The `up` guard matches `on()`, since the aggregated
result has no `instance` any more. A new report still fires with the
same value and stays for 15 minutes, at most a minute later than before
because of the one-minute step. The alerts now carry `hostname` and
`severity`, and no longer the controller's `instance`, `job` and
`controller_node`, the `module`, or the external labels of the scrape.

`linstorControllerErrorRate` gets the same form. Its `hostname` is the
controller pod name and the piraeus controller keeps its reports in an
[`emptyDir`](https://github.com/piraeusdatastore/piraeus-operator/blob/v2.10.2/pkg/resources/cluster/controller/controller-deployment.yaml#L111-L112),
so a new pod starts with no reports and a move alone does not trip it,
but a label change does whenever the controller has reports.

A gap longer than the hour is not covered. After it the series starts
over, and as before the `up` guard holds the alert back only if the
controller target recorded failed scrapes during the gap.

I checked it against the stored metrics of the cluster above with
read-only queries to vmselect. Over the upgrade the current expression
returns the three nodes from 15:45:30 to 16:00:00 UTC at a 30-second
step, the same interval as vmalert's `ALERTS` series, and over the label
change from 20:27:30 to 20:42:00. The new expression returns nothing
over both windows, and without `> 0` it returns each of the three nodes
with 0 at every step, 88 and 121 points. The same storage has six gaps
of 8 to 113 minutes over one night with no failed scrape recorded in
them. Replaying vmalert's instant query of the `increase()` part every
30 seconds in the 154 evaluations after those gaps, the current
expression returns at least one node in 99, the `hostname` key without
the hold in 84, and the new expression in 30, all of them after the
113-minute gap. Retention there does not reach back to a real new
report, so the firing side is covered by the tests.

`hack/linstor-error-report-alerts.bats` replays both rules through
`vmalert-tool` v1.136.0, pinned by digest, which is the VictoriaMetrics
release victoria-metrics-operator v0.68.4 [deploys by
default](https://github.com/VictoriaMetrics/operator/blob/v0.68.4/internal/config/config.go#L38),
with 30-second samples and 15-second evaluations like the platform's
vmagent and vmalert. The suite needs docker, like the migration suites,
and only the image pull touches the network.
`hack/testdata/linstor-error-report-alerts/unittest.yaml` has 20 cases
in five groups, which cover a controller move followed by a new report,
a changed external label for both rules, one satellite away for 40
minutes and one that comes back with a new report, a new report and a
first report on a node for both rules, and a controller that cannot be
scraped for 70 minutes. Against the rule on main 13 cases fail, 9 of
them because old reports fire again after the move, the label change or
the disconnect, and the other 4 on the label set alone, since the
current alerts carry the controller's labels. Each of seven mutations of
the new expressions turns at least one case red, namely no hold, a
15-minute hold, the guard on `instance`, no guard, the key on `hostname`
and `instance`, the key on `hostname` and `cluster`, and the controller
rule as on main. With Bats 1.14.0 at the commit CI pins, the new suite
with `alert-severity-contract`, `bats-strict-setup`,
`bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage`
and `image-pin-consistency` passes 74 of 74, and `helm template` renders
the same 15 rules as main. I have not run the new rules in vmalert on a
live cluster.

The commit cherry-picks onto release-1.7 as is, and the suite passes
there. On release-1.6 it applies without conflicts, but that branch runs
`hack/*.bats` through `hack/cozytest.sh` and has no
`hack/test_helper.bash`, so the suite there has to drop its `load
test_helper` line.

### Screenshots

Not a UI change.

### Downstream repositories

A rule file under `packages/system/piraeus-operator/alerts/`, a new Bats
suite under `hack/` and its test data. No values, schema, README, image,
make target or `ApplicationDefinition` change, and the website does not
mention these rules or the metric, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(piraeus-operator): the LINSTOR error-report alerts no longer fire on old reports when the LINSTOR controller pod moves to another node, when an external label of the scrape changes, or when a satellite reconnects within an hour. They now count reports per node, and their alerts carry the node's hostname instead of the controller pod's instance, job and controller_node labels.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved LINSTOR controller and satellite error alerts to account for
recent error reports over a 15-minute window, even when metrics are
intermittent or scrape labels change.
* Updated alert matching to better handle controller moves between nodes
and satellites that disappear from metrics while disconnected.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 18:18:47 +03:00
2025-10-08 09:43:34 +05:00
2024-02-08 12:04:32 +01:00

Cozystack Cozystack

GitHub Release Apache-2.0 License Commit activity OpenSSF Best Practices OpenSSF Scorecard CLOMonitor Support

Cozystack

Cozystack is a free and open-source platform and framework for building clouds.

Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.

With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.

Use Cozystack to build your own cloud or provide a cost-effective development environment.

Cozystack user interface

Use-Cases

Documentation

The documentation is located on the cozystack.io website.

Read the Getting Started section for a quick start.

If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.

Versioning

Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.

Contributions

Contributions are highly appreciated and very welcomed!

In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.

You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.

If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.

Community

You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.

Project documents

Document What it covers
Governance How decisions are made and who makes them
Contributor Ladder How to grow from contributor to maintainer
Maintainers Who maintains the project
Code of Conduct What we expect of each other
Security Policy Reporting a vulnerability, and which releases get fixes
Adopters Who runs Cozystack in production
Roadmap Where the project is going

License

Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.

Commercial Support

A list of companies providing commercial support for this project can be found on official site.

Languages
Go 52%
Shell 34%
TypeScript 6.8%
Go Template 4.7%
Makefile 1.3%
Other 1.1%