Three defects, found together because they are the same failure wearing
three hats: monitoring that looks configured and is not.
Fleet alert thresholds were never evaluated. check_fleet_thresholds was
written, was correct, and had no caller anywhere in the backend, so a
threshold set on the Fleet page was checked against nothing and a server
could sit at 99% CPU for a week in silence. It now runs every minute as a
built-in scheduled job alongside every other recurring task, and each
alert it opens or resolves goes to admins through the Notification Bus.
An evaluation nobody is told about is most of the way to no monitoring.
The agent heartbeat was throwing metrics away. It built its own
ServerMetrics row and filled five of the model's thirteen columns,
dropping memory_used, disk_used, the network totals, both network rates
and extra. The two rates are exactly what the threshold checker reads, so
a network threshold could never fire for an agent-reported server: the
data needed to evaluate it was discarded on the way in. Heartbeats now go
through ServerMetricsService, the writer everything else already used.
Metric history outlived its own retention. The history and aggregate
endpoints offered 7-day and 30-day windows while cleanup deleted every
sample after seven days, and two constants described hourly and daily
aggregate retention that nothing produced. A 30-day chart came back
holding at most seven days and said nothing about it. Both endpoints now
return a coverage block — the retention in force, the earliest point they
can answer for, and whether the window asked for was truncated — and the
two constants that promised retention nobody implemented are gone. Real
rolled-up aggregates are the other way to fix this and would need their
own tables and writer; until they exist, saying what we have is the
honest answer.
Two more things surfaced while writing the tests.
get_aggregated_metrics grouped by date_trunc, which is PostgreSQL's alone,
so the endpoint raised "no such function: date_trunc" on every SQLite
deployment — one of the two supported databases. Buckets are now built
with the right function for the database in use.
And an escalation is not a recovery. warning -> critical closes the
warning row, so both a genuine recovery and a worsening breach end up as
status='resolved'; announcing the second as "recovered" would tell an
operator the problem went away at the moment it got worse. Supersession is
tracked separately and is not notified twice, because the new alert
already says what happened.
The two built-in-job counts in test_jobs.py move from 16 to 17 and 24 to
25, and BASELINE_COUNT is raised for the new tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
