fix(monitoring-agents): make a silent Watchdog an alarm in Alerta

Alerta turns the Watchdog into a heartbeat and stores its severity in the
heartbeat attributes. The alerta heartbeats --alert job of the alerta-web
image takes the severity of a stale heartbeat from those attributes ahead
of its HEARTBEAT_SEVERITY default, so with severity ok the HeartbeatFail
it raises is a normal alert and nothing says the alerting pipeline
stopped. Without a timeout label the heartbeat also took a day to go
stale, while Alertmanager refreshes it every minute.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
This commit is contained in:
Yan Bondarenko committed 2026-10-25 17:15:04 +03:00
1 parent 55aad6f914
commit a7ec05f9eb
2 files changed
+31 -1

No files matched your search

@@ -33,7 +33,8 @@ spec:
is working properly.
expr: vector(1)
labels:
severity: ok
severity: major
timeout: "300"
exported_instance: global
service: general.rules
event: Heartbeat
@@ -0,0 +1,29 @@
suite: Watchdog heartbeat raises an alarm in Alerta when it stops
templates:
- templates/alerts.yaml
release:
name: monitoring-agents
namespace: cozy-monitoring
tests:
- it: gives the heartbeat a severity Alerta reports as an alarm
documentSelector:
path: metadata.name
value: alerts-general.rules
asserts:
- equal:
path: spec.groups[0].rules[?(@.alert == "Watchdog")].labels.severity
value: major
- it: lets the heartbeat go stale within minutes, not a day
documentSelector:
path: metadata.name
value: alerts-general.rules
asserts:
- equal:
path: spec.groups[0].rules[?(@.alert == "Watchdog")].labels.timeout
value: "300"
- equal:
path: spec.groups[0].rules[?(@.alert == "Watchdog")].labels.event
value: Heartbeat