mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-22 20:53:52 +09:00
fix(monitoring-agents): make a silent Watchdog an alarm in Alerta
Alerta turns the Watchdog into a heartbeat and stores its severity in the heartbeat attributes. The alerta heartbeats --alert job of the alerta-web image takes the severity of a stale heartbeat from those attributes ahead of its HEARTBEAT_SEVERITY default, so with severity ok the HeartbeatFail it raises is a normal alert and nothing says the alerting pipeline stopped. Without a timeout label the heartbeat also took a day to go stale, while Alertmanager refreshes it every minute. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
This commit is contained in:
1 parent
55aad6f914
commit
a7ec05f9eb
2 files changed
+31
-1
No files matched your search
@@ -33,7 +33,8 @@ spec:
|
||||
is working properly.
|
||||
expr: vector(1)
|
||||
labels:
|
||||
severity: ok
|
||||
severity: major
|
||||
timeout: "300"
|
||||
exported_instance: global
|
||||
service: general.rules
|
||||
event: Heartbeat
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
suite: Watchdog heartbeat raises an alarm in Alerta when it stops
|
||||
templates:
|
||||
- templates/alerts.yaml
|
||||
|
||||
release:
|
||||
name: monitoring-agents
|
||||
namespace: cozy-monitoring
|
||||
|
||||
tests:
|
||||
- it: gives the heartbeat a severity Alerta reports as an alarm
|
||||
documentSelector:
|
||||
path: metadata.name
|
||||
value: alerts-general.rules
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.groups[0].rules[?(@.alert == "Watchdog")].labels.severity
|
||||
value: major
|
||||
|
||||
- it: lets the heartbeat go stale within minutes, not a day
|
||||
documentSelector:
|
||||
path: metadata.name
|
||||
value: alerts-general.rules
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.groups[0].rules[?(@.alert == "Watchdog")].labels.timeout
|
||||
value: "300"
|
||||
- equal:
|
||||
path: spec.groups[0].rules[?(@.alert == "Watchdog")].labels.event
|
||||
value: Heartbeat
|
||||
Reference in new issue
Block a user