## What this PR does
The Watchdog is meant to prove the alerting pipeline is alive, and
Alerta turns it into a heartbeat because it carries `event: Heartbeat`.
When the heartbeat goes stale, nothing alarms.
- The Alerta 9.1.0 heartbeat plugin (`alerta/plugins/heartbeat.py`)
stores the alert severity in the heartbeat attributes, and the Watchdog
has `severity: ok`.
- The `alerta/alerta-web:9.1.0` image sets `HEARTBEAT_SEVERITY=major`,
and supervisord runs `alerta heartbeats --alert --severity major` about
every minute. The client it ships, 8.5.3, takes the severity of a stale
heartbeat from the heartbeat attributes ahead of that flag
(`want_severity = b.attributes.pop('severity', severity)` in
`alertaclient/commands/cmd_heartbeats.py`). The HeartbeatFail it raises
for the Watchdog is therefore severity `ok`, a normal alert.
- The Watchdog has no `timeout` label, so the Alerta webhook leaves the
timeout unset and the heartbeat only goes stale after a day (timeout
86400 on three v1.6.5 clusters), while the `event="Heartbeat"` route
resends it every minute. `/api/management/healthcheck` compares against
four timeouts and never reports it.
The Watchdog now has `severity: major`, matching the image's own
`HEARTBEAT_SEVERITY`, and `timeout: "300"`, five missed resends. The
`event="Heartbeat"` route still sends it only to `alerta_no_resolve` and
stops, so the severity reaches no email or other receiver, and the
template has no inhibit rules it could trigger. In the Alertmanager and
vmalert lists the Watchdog now shows as major instead of ok.
A new suite `tests/watchdog_heartbeat_test.yaml` checks the severity,
the timeout and the event label. `helm unittest .` in
`packages/system/monitoring-agents` passes 21 of 21 with helm-unittest
1.1.2, and against the rules on main the two new cases fail and the
other 19 pass. `hack/alert-severity-contract.bats` passes 9 of 9. I have
not run the change on a live cluster, and I have not read a heartbeat
record through the Alerta API, the chain above is from the sources of
the versions the chart pins.
The rule file is the same on release-1.6, and that backport is a
separate PR.
### Screenshots
Not a UI change.
### Downstream repositories
One rule in
`packages/system/monitoring-agents/alerts/general.rules.yaml` and a
test. No values key, schema, default or package change, so no trigger
map entry applies.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(monitoring-agents): a Watchdog that stops reaching Alerta now raises a major HeartbeatFail about six minutes later. It used to raise an ok alert after a day.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Monitoring**
* The Watchdog alert now reports a major severity and includes a
300-second timeout.
* The alert identifies heartbeat events.
* **Tests**
* Added checks to verify the Watchdog alert’s severity, timeout, and
event.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Cozystack
Cozystack is a free and open-source platform and framework for building clouds.
Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack to build your own platform
You can install only the Cozystack components you need on an existing Kubernetes cluster
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.
Project documents
| Document | What it covers |
|---|---|
| Governance | How decisions are made and who makes them |
| Contributor Ladder | How to grow from contributor to maintainer |
| Maintainers | Who maintains the project |
| Code of Conduct | What we expect of each other |
| Security Policy | Reporting a vulnerability, and which releases get fixes |
| Adopters | Who runs Cozystack in production |
| Roadmap | Where the project is going |
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.
