fix(monitoring): route InfoInhibitor to the blackhole receiver (#4948)

## What this PR does

`InfoInhibitor` in
`packages/system/monitoring-agents/alerts/general.rules.yaml` is the
kube-prometheus rule that exists only to feed an Alertmanager inhibit
rule, and its own description says it should go to a null receiver. The
`update` target that first imported the rules rewrote its `severity:
none` to `major`, and the platform Alertmanager config has no inhibit
rule, so the alert goes to Alerta one level below critical. Alerta's
Telegram and Slack plugins forward it with the default
`disabledSeverity: []`, and once email delivery is configured, a
`notify_email="true"` label inherited from the info alert gets it
emailed as well.

It fires whenever an alert with `severity="info"` is active and no
warning or critical alert fires in the same namespace and cluster. #3799
moved the last first-party rule off `info`, but rules the platform does
not write still use it. CDI v1.66.1 generates
`CDIStorageProfilesIncomplete`, KubeVirt v1.9.0 generates
`KubeVirtVMGuestMemoryAvailableLow` and
`KubeVirtDeprecatedAPIRequested`, all at `info`, and operators add their
own. We hit it on a v1.6.4 cluster with one custom `severity: info` rule
that is always firing, and Alerta showed a permanent MAJOR
`InfoInhibitor`. The info alert itself cannot show up there, because
Alerta's default alarm model has no `info`.

This puts the label back to `none`, the value kube-prometheus ships. The
platform Alertmanager config already routes `severity="none"` to the
`blackhole` receiver ahead of every route that delivers, and the
Monitoring parameters already document that such alerts never reach
email. `hack/alert-severity-contract.bats` refuses `none` on purpose,
because an alert carrying it is never delivered, so InfoInhibitor is
listed as the one alert for which that is the point. The exemption
covers `none` and that name only.

The contract now also checks that InfoInhibitor carries `none`, that
`none` still fails on any other alert, and that the exemption does not
let InfoInhibitor through with a severity Alerta refuses. A new
helm-unittest suite in `packages/system/monitoring` pins the route
order, so that only the heartbeat route can come ahead of the blackhole,
with and without the email routes.

There is no inhibit rule here, and the expression still matches `info`
only. Wiring the inhibition up would hide every informational alert from
Alerta unless a warning fires in the same namespace, which changes
behaviour on every cluster and is a separate decision. Dropping
InfoInhibitor altogether would be equivalent for delivery, and I can
rework the PR that way if you prefer it.

v1.6.5 ships the same rule and the same routes, so a backport to
`release-1.6` would be appreciated. The commit cherry-picks onto it
without conflicts.

### Validation

- `bats hack/alert-severity-contract.bats` passes 12 of 12, and so does
the same file under `hack/cozytest.sh`. Without the label change the new
InfoInhibitor test fails on `[ "$delivered" -eq 0 ]`. With the exemption
removed the tree-wide contract fails on
`packages/system/monitoring-agents/alerts/general.rules.yaml:
InfoInhibitor -> 'none'`. With the exemption widened to any alert, or to
any severity, the matching fixture test fails.
- `helm unittest` in `packages/system/monitoring` passes 25 suites and
306 tests. Deleting the `severity="none"` route fails both new cases,
and an email route placed ahead of it fails the email case.
- `helm unittest` in `packages/system/monitoring-agents` passes 6 suites
and 19 tests, and the rendered `alerts-general.rules` VMRule carries
`severity: none` for InfoInhibitor.
- Rendered the Alertmanager config and ran `amtool config routes test`
from Alertmanager v0.31.0, the default of VM operator v0.68.4, with
InfoInhibitor's labels. `severity=major` routes to `alerta`, and to
`alerta,email` once email is configured and `notify_email=true` is
inherited. `severity=none` routes to `blackhole` in every case. `amtool
check-config` reports 0 inhibit rules.
- On `release-1.6` with this commit cherry-picked, the contract and both
helm-unittest suites pass.
- Not exercised on a live cluster.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Walked the trigger map against the diff. It changes one alert label
under `packages/system/monitoring-agents/alerts/`, extends the test in
`hack/alert-severity-contract.bats` and adds a test suite under
`packages/system/monitoring/tests/`. No package is added or renamed, and
no values, schema, CRD, platform values, variant, make target, tool the
satellites call, node prerequisite or metric changes. The website
already says that `severity="none"` alerts are blackholed and never
reach email, and no page mentions InfoInhibitor, so nothing there goes
stale.

### Release note

```release-note
fix(monitoring): InfoInhibitor no longer reaches Alerta as a major alert. It carries severity none again, which the platform Alertmanager routes to the blackhole, as upstream kube-prometheus intends. Before, any active alert with severity info, from a custom rule or from one CDI or KubeVirt generates, raised a major InfoInhibitor in Alerta and in its Telegram and Slack notifications.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* `InfoInhibitor` alerts are now treated as non-notifying, preventing
them from generating alert notifications. Other alerts retain their
existing severity handling, and heartbeat alerts continue to follow
their configured route.
* The non-notifying route takes precedence over email delivery rules
that include `InfoInhibitor`.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
Aleksei Sviridkin authored and GitHub committed 2026-11-13 15:56:58 +03:00
commit 185c5eca72
3 files changed
+141 -1

No files matched your search

+89
View File
@@ -9,6 +9,10 @@ load test_helper
# scope: they are re-vendored by `make update`, so a bad severity there is
# fixed upstream or in patches/, not by editing the label.
ALERT_SEVERITY_ALLOWED=" security critical major minor warning indeterminate informational normal ok cleared debug trace unknown "
# The alerts that exist only to be dropped, the one place `none` is right.
# InfoInhibitor is the source side of an Alertmanager inhibit rule, which the
# platform config does not have, so delivering it at any severity is noise.
ALERT_SEVERITY_BLACKHOLED=" InfoInhibitor "
helm_actions_stripped() {
awk '
@@ -104,6 +108,14 @@ alert_severity_contract() {
break
fi
done
if [ "$severity" = "none" ]; then
for blackholed in $ALERT_SEVERITY_BLACKHOLED; do
if [ "$blackholed" = "$alert" ]; then
accepted=1
break
fi
done
fi
if [ "$accepted" -eq 0 ]; then
echo "$file: $alert -> '$severity'" >&2
bad=$((bad + 1))
@@ -128,6 +140,83 @@ alert_severity_contract() {
alert_severity_contract
}
@test "InfoInhibitor is labelled for the blackhole route" {
rows=$(mktemp) || return 1
if ! alert_severity_rows > "$rows"; then
rm -f "$rows"
return 1
fi
tab=$(printf '\t')
found=$(grep -c "${tab}InfoInhibitor${tab}" "$rows" || true)
delivered=$(grep "${tab}InfoInhibitor${tab}" "$rows" | grep -vc "${tab}none\$" || true)
rm -f "$rows"
[ "$found" -gt 0 ]
[ "$delivered" -eq 0 ]
}
@test "severity none passes only on an alert that exists to be dropped" {
fixture=$(mktemp -d)
mkdir -p "$fixture/packages/system/example/alerts"
printf '%s\n' \
'apiVersion: operator.victoriametrics.com/v1beta1' \
'kind: VMRule' \
'spec:' \
' groups:' \
' - name: general' \
' rules:' \
' - alert: InfoInhibitor' \
' labels:' \
' severity: none' \
> "$fixture/packages/system/example/alerts/00-inhibitor.yaml"
git -C "$fixture" init -q
git -C "$fixture" add packages/system/example/alerts/00-inhibitor.yaml
inhibitor_succeeded=0
ALERT_SEVERITY_ROOT="$fixture" alert_severity_contract >/dev/null 2>&1 || inhibitor_succeeded=$?
printf '%s\n' \
'apiVersion: operator.victoriametrics.com/v1beta1' \
'kind: VMRule' \
'spec:' \
' groups:' \
' - name: silent' \
' rules:' \
' - alert: SilentAlert' \
' labels:' \
' severity: none' \
> "$fixture/packages/system/example/alerts/01-silent.yaml"
git -C "$fixture" add packages/system/example/alerts/01-silent.yaml
silent_succeeded=0
ALERT_SEVERITY_ROOT="$fixture" alert_severity_contract >/dev/null 2>&1 || silent_succeeded=$?
rm -rf "$fixture"
[ "$inhibitor_succeeded" -eq 0 ]
[ "$silent_succeeded" -ne 0 ]
}
@test "an alert that exists to be dropped still fails on a refused severity" {
fixture=$(mktemp -d)
mkdir -p "$fixture/packages/system/example/alerts"
printf '%s\n' \
'apiVersion: operator.victoriametrics.com/v1beta1' \
'kind: VMRule' \
'spec:' \
' groups:' \
' - name: general' \
' rules:' \
' - alert: InfoInhibitor' \
' labels:' \
' severity: info' \
> "$fixture/packages/system/example/alerts/00-inhibitor.yaml"
git -C "$fixture" init -q
git -C "$fixture" add packages/system/example/alerts/00-inhibitor.yaml
contract_succeeded=0
ALERT_SEVERITY_ROOT="$fixture" alert_severity_contract >/dev/null 2>&1 || contract_succeeded=$?
rm -rf "$fixture"
[ "$contract_succeeded" -ne 0 ]
}
@test "a parser failure cannot leave the contract green" {
fixture=$(mktemp -d)
mkdir -p "$fixture/packages/system/example/alerts"
@@ -53,6 +53,6 @@ spec:
!= "InfoInhibitor", severity =~ "warning|critical", alertstate="firing"} ==
1
labels:
severity: major
severity: none
exported_instance: global
service: general.rules
@@ -0,0 +1,51 @@
suite: Alertmanager blackhole route
templates:
- templates/alerta/alerta.yaml
release:
name: monitoring
namespace: tenant-root
set:
_cluster:
root-host: example.org
_namespace:
host: tenant-root.example.org
ingress: tenant-root
# Rules labelled severity="none", InfoInhibitor among them, exist to be
# dropped, and Alerta answers that severity with a 500. Only the heartbeat
# route may come ahead of the blackhole, so no delivering route claims them.
tests:
- it: drops severity none ahead of every delivering route
asserts:
- documentSelector:
path: metadata.labels.app
value: alertmanager
matchRegex:
path: stringData["alertmanager.yaml"]
pattern: '\n routes:\n - receiver: alerta_no_resolve\n repeat_interval: 1m\n matchers:\n - .event="Heartbeat".\n - receiver: blackhole\n matchers:\n - .severity="none".\n'
- it: keeps severity none out of email even when its alert name is listed
set:
alerta:
alerts:
email:
to: "oncall@example.org"
from: "alerts@example.org"
smarthost: "smtp.example.org:465"
alertnames:
- InfoInhibitor
asserts:
- documentSelector:
path: metadata.labels.app
value: alertmanager
matchRegex:
path: stringData["alertmanager.yaml"]
pattern: '\n routes:\n - receiver: alerta_no_resolve\n repeat_interval: 1m\n matchers:\n - .event="Heartbeat".\n - receiver: blackhole\n matchers:\n - .severity="none".\n'
- documentSelector:
path: metadata.labels.app
value: alertmanager
matchRegex:
path: stringData["alertmanager.yaml"]
pattern: '(?s)\n - receiver: blackhole\n matchers:\n - .severity="none".\n.*\n - receiver: email\n group_wait: 1m\n group_interval: 5m\n repeat_interval: .12h.\n matchers:\n - .alertname=~"\^\(InfoInhibitor\)\$".\n'