mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
## What this PR does `InfoInhibitor` in `packages/system/monitoring-agents/alerts/general.rules.yaml` is the kube-prometheus rule that exists only to feed an Alertmanager inhibit rule, and its own description says it should go to a null receiver. The `update` target that first imported the rules rewrote its `severity: none` to `major`, and the platform Alertmanager config has no inhibit rule, so the alert goes to Alerta one level below critical. Alerta's Telegram and Slack plugins forward it with the default `disabledSeverity: []`, and once email delivery is configured, a `notify_email="true"` label inherited from the info alert gets it emailed as well. It fires whenever an alert with `severity="info"` is active and no warning or critical alert fires in the same namespace and cluster. #3799 moved the last first-party rule off `info`, but rules the platform does not write still use it. CDI v1.66.1 generates `CDIStorageProfilesIncomplete`, KubeVirt v1.9.0 generates `KubeVirtVMGuestMemoryAvailableLow` and `KubeVirtDeprecatedAPIRequested`, all at `info`, and operators add their own. We hit it on a v1.6.4 cluster with one custom `severity: info` rule that is always firing, and Alerta showed a permanent MAJOR `InfoInhibitor`. The info alert itself cannot show up there, because Alerta's default alarm model has no `info`. This puts the label back to `none`, the value kube-prometheus ships. The platform Alertmanager config already routes `severity="none"` to the `blackhole` receiver ahead of every route that delivers, and the Monitoring parameters already document that such alerts never reach email. `hack/alert-severity-contract.bats` refuses `none` on purpose, because an alert carrying it is never delivered, so InfoInhibitor is listed as the one alert for which that is the point. The exemption covers `none` and that name only. The contract now also checks that InfoInhibitor carries `none`, that `none` still fails on any other alert, and that the exemption does not let InfoInhibitor through with a severity Alerta refuses. A new helm-unittest suite in `packages/system/monitoring` pins the route order, so that only the heartbeat route can come ahead of the blackhole, with and without the email routes. There is no inhibit rule here, and the expression still matches `info` only. Wiring the inhibition up would hide every informational alert from Alerta unless a warning fires in the same namespace, which changes behaviour on every cluster and is a separate decision. Dropping InfoInhibitor altogether would be equivalent for delivery, and I can rework the PR that way if you prefer it. v1.6.5 ships the same rule and the same routes, so a backport to `release-1.6` would be appreciated. The commit cherry-picks onto it without conflicts. ### Validation - `bats hack/alert-severity-contract.bats` passes 12 of 12, and so does the same file under `hack/cozytest.sh`. Without the label change the new InfoInhibitor test fails on `[ "$delivered" -eq 0 ]`. With the exemption removed the tree-wide contract fails on `packages/system/monitoring-agents/alerts/general.rules.yaml: InfoInhibitor -> 'none'`. With the exemption widened to any alert, or to any severity, the matching fixture test fails. - `helm unittest` in `packages/system/monitoring` passes 25 suites and 306 tests. Deleting the `severity="none"` route fails both new cases, and an email route placed ahead of it fails the email case. - `helm unittest` in `packages/system/monitoring-agents` passes 6 suites and 19 tests, and the rendered `alerts-general.rules` VMRule carries `severity: none` for InfoInhibitor. - Rendered the Alertmanager config and ran `amtool config routes test` from Alertmanager v0.31.0, the default of VM operator v0.68.4, with InfoInhibitor's labels. `severity=major` routes to `alerta`, and to `alerta,email` once email is configured and `notify_email=true` is inherited. `severity=none` routes to `blackhole` in every case. `amtool check-config` reports 0 inhibit rules. - On `release-1.6` with this commit cherry-picked, the contract and both helm-unittest suites pass. - Not exercised on a live cluster. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Walked the trigger map against the diff. It changes one alert label under `packages/system/monitoring-agents/alerts/`, extends the test in `hack/alert-severity-contract.bats` and adds a test suite under `packages/system/monitoring/tests/`. No package is added or renamed, and no values, schema, CRD, platform values, variant, make target, tool the satellites call, node prerequisite or metric changes. The website already says that `severity="none"` alerts are blackholed and never reach email, and no page mentions InfoInhibitor, so nothing there goes stale. ### Release note ```release-note fix(monitoring): InfoInhibitor no longer reaches Alerta as a major alert. It carries severity none again, which the platform Alertmanager routes to the blackhole, as upstream kube-prometheus intends. Before, any active alert with severity info, from a custom rule or from one CDI or KubeVirt generates, raised a major InfoInhibitor in Alerta and in its Telegram and Slack notifications. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * `InfoInhibitor` alerts are now treated as non-notifying, preventing them from generating alert notifications. Other alerts retain their existing severity handling, and heartbeat alerts continue to follow their configured route. * The non-notifying route takes precedence over email delivery rules that include `InfoInhibitor`. <!-- end of auto-generated comment: release notes by coderabbit.ai -->