mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
## What this PR does
The Watchdog is meant to prove the alerting pipeline is alive, and
Alerta turns it into a heartbeat because it carries `event: Heartbeat`.
When the heartbeat goes stale, nothing alarms.
- The Alerta 9.1.0 heartbeat plugin (`alerta/plugins/heartbeat.py`)
stores the alert severity in the heartbeat attributes, and the Watchdog
has `severity: ok`.
- The `alerta/alerta-web:9.1.0` image sets `HEARTBEAT_SEVERITY=major`,
and supervisord runs `alerta heartbeats --alert --severity major` about
every minute. The client it ships, 8.5.3, takes the severity of a stale
heartbeat from the heartbeat attributes ahead of that flag
(`want_severity = b.attributes.pop('severity', severity)` in
`alertaclient/commands/cmd_heartbeats.py`). The HeartbeatFail it raises
for the Watchdog is therefore severity `ok`, a normal alert.
- The Watchdog has no `timeout` label, so the Alerta webhook leaves the
timeout unset and the heartbeat only goes stale after a day (timeout
86400 on three v1.6.5 clusters), while the `event="Heartbeat"` route
resends it every minute. `/api/management/healthcheck` compares against
four timeouts and never reports it.
The Watchdog now has `severity: major`, matching the image's own
`HEARTBEAT_SEVERITY`, and `timeout: "300"`, five missed resends. The
`event="Heartbeat"` route still sends it only to `alerta_no_resolve` and
stops, so the severity reaches no email or other receiver, and the
template has no inhibit rules it could trigger. In the Alertmanager and
vmalert lists the Watchdog now shows as major instead of ok.
A new suite `tests/watchdog_heartbeat_test.yaml` checks the severity,
the timeout and the event label. `helm unittest .` in
`packages/system/monitoring-agents` passes 21 of 21 with helm-unittest
1.1.2, and against the rules on main the two new cases fail and the
other 19 pass. `hack/alert-severity-contract.bats` passes 9 of 9. I have
not run the change on a live cluster, and I have not read a heartbeat
record through the Alerta API, the chain above is from the sources of
the versions the chart pins.
The rule file is the same on release-1.6, and that backport is a
separate PR.
### Screenshots
Not a UI change.
### Downstream repositories
One rule in
`packages/system/monitoring-agents/alerts/general.rules.yaml` and a
test. No values key, schema, default or package change, so no trigger
map entry applies.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(monitoring-agents): a Watchdog that stops reaching Alerta now raises a major HeartbeatFail about six minutes later. It used to raise an ok alert after a day.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Monitoring**
* The Watchdog alert now reports a major severity and includes a
300-second timeout.
* The alert identifies heartbeat events.
* **Tests**
* Added checks to verify the Watchdog alert’s severity, timeout, and
event.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->