fix(monitoring): send alerts to every Alertmanager replica (#4889)

## What this PR does

vmalert sends every alert to every Alertmanager replica instead of to
the shared Service.

The chart gave vmalert a single notifier,
`http://vmalertmanager-alertmanager.<namespace>.svc:9093`. That Service
is headless and resolves to the three Alertmanager pods, but each alert
reached one replica at a time and moved to another one whenever vmalert
reconnected. The Alertmanager README asks for the opposite. It says "Do
not load balance traffic between Prometheus and its Alertmanagers" and
to point the sender at every instance, because the cluster deduplicates
notifications only when each replica receives each alert.

What this looked like on three clusters running 1.6.4, read through the
Alertmanager API of each pod.

- On all three, every active alert sat on one replica and the other two
had none. The counts were 0, 0 and 10 on the first, 0, 0 and 23 on the
second, and 1, 0 and 9 on the third.
- On the third one, which also has an email receiver, the active alert
count of each replica changed 163 to 180 times in 24 hours. The three
replicas sent 63, 67 and 62 emails in that window. Over the same week
only three alert names with a severity routed to email fired there.

An alert that leaves a replica expires there and goes out as resolved,
and the replica that receives it next sends it as a new firing
notification. So a single long-running alert keeps producing firing and
resolved notifications, and a receiver that rejects concurrent
duplicates sees them as conflicts.

The change.

- `vm/vmalert.yaml` renders one notifier per Alertmanager pod,
`http://vmalertmanager-alertmanager-<i>.vmalertmanager-alertmanager.<namespace>.svc:9093`.
That is the address the operator builds for StatefulSet pods
(`VMAlertmanager.asPodFQDN`), and the replicas already reach each other
through the same per-pod names in their `--cluster.peer` arguments.
- The replica count comes from a new `monitoring.alertmanager.replicas`
helper, which now also sets `spec.replicaCount` of the VMAlertmanager.
The notifier list cannot drift from the number of pods.

Why not the operator's notifier selector. `VMAlertNotifierSpec.selector`
expands matched VMAlertmanagers into the same per-pod addresses, but
only when the VMAlert is reconciled. The VMAlert controller does not
watch VMAlertmanager objects, so a VMAlert reconciled before its
VMAlertmanager exists, which can happen on a fresh install, would get no
notifier until something else triggers a reconcile. A static list
renders the same addresses without depending on that order.

Upgrade impact. The VMAlertmanager is unchanged, it renders
`replicaCount: 3` as before. The VMAlert gets new notifier arguments, so
the operator restarts the vmalert pod once.

Testing.

- `helm unittest` passes for the monitoring chart, 24 suites and 304
tests. The new suite `tests/vmalert_notifiers_test.yaml` checks the
three notifier addresses, the replica count of the VMAlertmanager and
the release namespace in the addresses.
- Each of these breakages turns at least one case of the new suite red.
The notifier back on the Service address, the helper set to 2, a
hardcoded namespace in the address, and a hardcoded `replicaCount` that
no longer follows the helper.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Walked the trigger map in `docs/agents/contributing.md` against the
diff. It changes two templates and the helpers of
`packages/system/monitoring` and adds a test file. No values, schema,
README, package name, make target or `ApplicationDefinition` changes, so
no entry applies.

### Release note

```release-note
fix(monitoring): vmalert now sends alerts to every Alertmanager replica instead of the shared Service, so the Alertmanager cluster deduplicates them and a long-running alert no longer produces repeated firing and resolved notifications.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Improvements**
* vmalert now configures a notifier URL for each of the three
Alertmanager replicas, rather than using a single shared URL.
* Alertmanager’s replica count is managed consistently with the notifier
configuration.
* **Tests**
* Added checks for the replica count, all three notifier URLs, and the
configured namespace.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

packages/system/monitoring/templates/_helpers.tpl

@@ -327,3 +327,12 @@
 {{- define "monitoring.tracingCentral" -}}
 {{- if and (ne .Release.Namespace "tenant-root") (eq (toString (dig "tracingCentral" false (.Values._namespace | default dict))) "true") -}}true{{- end -}}
 {{- end -}}
+
+{{- /*
+  Alertmanager replica count. vmalert lists each replica as its own
+  notifier from the same number, because the Alertmanager cluster only
+  deduplicates notifications when every replica receives every alert.
+*/}}
+{{- define "monitoring.alertmanager.replicas" -}}
+3
+{{- end -}}

packages/system/monitoring/templates/alerta/alerta.yaml

@@ -419,7 +419,7 @@ spec:
       apps.cozystack.io/application.group: apps.cozystack.io
       apps.cozystack.io/application.kind: Monitoring
       apps.cozystack.io/application.name: {{ $.Release.Name }}
-  replicaCount: 3
+  replicaCount: {{ include "monitoring.alertmanager.replicas" $ }}
   configSecret: alertmanager
   {{- if and $emailEnabled $email.authUsername }}
   secrets:

packages/system/monitoring/templates/vm/vmalert.yaml

@@ -16,7 +16,9 @@ spec:
   extraArgs:
     remoteWrite.disablePathAppend: "true"
   notifiers:
-  - url: http://vmalertmanager-alertmanager.{{ $.Release.Namespace }}.svc:9093
+  {{- range $i := until (int (include "monitoring.alertmanager.replicas" $)) }}
+  - url: http://vmalertmanager-alertmanager-{{ $i }}.vmalertmanager-alertmanager.{{ $.Release.Namespace }}.svc:9093
+  {{- end }}
   remoteRead:
     url: http://vmselect-{{ .name }}.{{ $.Release.Namespace }}.svc:8481/select/0/prometheus
   remoteWrite:

packages/system/monitoring/tests/vmalert_notifiers_test.yaml

@@ -0,0 +1,55 @@
+suite: vmalert notifies every Alertmanager replica
+templates:
+  - templates/vm/vmalert.yaml
+  - templates/alerta/alerta.yaml
+
+release:
+  name: monitoring
+  namespace: tenant-root
+
+set:
+  _cluster:
+    root-host: example.org
+  _namespace:
+    host: tenant-root.example.org
+    ingress: tenant-root
+  metricsStorages:
+    - name: shortterm
+      retentionPeriod: "3d"
+      deduplicationInterval: "15s"
+
+# The Alertmanager cluster deduplicates notifications only when every replica
+# receives every alert. A notifier pointing at the headless Service reaches
+# one pod per connection, so an alert lives on whichever replica that
+# connection landed on and moves when vmalert reconnects. Each replica it
+# leaves sends a resolved notification and each one it reaches sends a new
+# firing one.
+tests:
+  - it: lists each Alertmanager pod as a notifier instead of the shared Service
+    template: templates/vm/vmalert.yaml
+    asserts:
+      - equal:
+          path: spec.notifiers
+          value:
+            - url: http://vmalertmanager-alertmanager-0.vmalertmanager-alertmanager.tenant-root.svc:9093
+            - url: http://vmalertmanager-alertmanager-1.vmalertmanager-alertmanager.tenant-root.svc:9093
+            - url: http://vmalertmanager-alertmanager-2.vmalertmanager-alertmanager.tenant-root.svc:9093
+
+  - it: runs as many Alertmanager replicas as vmalert has notifiers
+    template: templates/alerta/alerta.yaml
+    documentSelector:
+      path: kind
+      value: VMAlertmanager
+    asserts:
+      - equal:
+          path: spec.replicaCount
+          value: 3
+
+  - it: addresses the Alertmanager pods in the release namespace
+    template: templates/vm/vmalert.yaml
+    release:
+      namespace: tenant-foo
+    asserts:
+      - equal:
+          path: spec.notifiers[2].url
+          value: http://vmalertmanager-alertmanager-2.vmalertmanager-alertmanager.tenant-foo.svc:9093