Files
cozystack/packages/system
scooby87 51155f1003 feat(postgres-operator): alert when the WAL archive backlog does not drain (#4572)
## What this PR does

`LastFailedArchiveTime` compares a cluster's last failed WAL archival
with its last successful one, so it resolves as soon as any segment gets
through. Archiving that fails intermittently, or runs slower than the
cluster writes WAL, keeps clearing that alert while the queue of
segments waiting on the Postgres volume grows, until the volume is full
and the database stops. The alert is also only a warning.

This PR adds `CNPGWALArchiveBacklog`, a critical alert that fires when a
primary's `archive_status` queue has not been empty once in 30 minutes.
A healthy archiver empties the queue within seconds of each WAL switch,
so a queue that never drains means point-in-time recovery is falling
behind, whatever the archiver reports. Idle clusters and replicas have
an empty queue and stay quiet.

Validated live, since the repository has no rule-evaluation tooling: on
a test cluster, two CNPG clusters archived into an object store that
refused connections and into one that accepted them and never answered,
while writing WAL continuously. The rule, evaluated by the root vmalert,
fired for both. Over the previous seven days of the same cluster's
metrics, none of its healthy clusters would have matched; those clusters
carry light load, so a cluster whose archiver runs near its limit all
the time has not been observed and would page, which is the intended
reading of a queue that never drains. The unit test pins only that the
rule ships in the rendered PrometheusRule as critical, since the rules
reach the cluster through a file glob that drops a moved file silently.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches an operator chart's alert rules and its test; no path
from the trigger map.

### Release note

```release-note
feat(postgres-operator): page when a Postgres cluster's WAL archive queue has not drained for 30 minutes, which the existing archiving alert misses while some archives still succeed.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added a critical alert for sustained PostgreSQL WAL archive backlogs.
It triggers when the minimum observed ready archive status remains above
zero over a 30-minute window, then stays pending for 10 minutes before
firing. The alert describes potential impacts on point-in-time recovery
and storage usage.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-30 21:21:22 +03:00
..
…
…
…