Aleksei Sviridkin 4fa6bfe30b feat(monitoring): alert when pods of one workload share a node (#4993)
## What this PR does

On three 3-node clusters running v1.6.4 and v1.6.5 we found a tenant
etcd with two of its three members on one node, a tenant Redis with two
of its three Sentinels on the node that also ran the master, and several
VictoriaMetrics and VictoriaLogs pairs (vmstorage, vmselect,
vmalertmanager, vlstorage) sharing a node. Nothing alerted. Losing one
of those nodes would have cost the etcd or Sentinel quorum, or both
copies of a storage tier. Spread constraints keep new pods apart once
they are in place (#4986 for etcd, #4989 for monitoring, #4990 for
Redis), but a placement that already exists, or that comes from a soft
rule or a drain, stays as it is, because the scheduler does not move
running pods.

This adds `PodReplicasColocated`, a warning in a new
`workload-placement` group under
`packages/system/monitoring-agents/alerts/`. It fires when two or more
running pods of one StatefulSet, ReplicaSet, CNPG `Cluster` or
`StrimziPodSet` have been on one node for an hour while the cluster has
at least as many schedulable nodes as the workload has running pods, so
five replicas on three nodes stay quiet. The alert carries `cluster`,
`namespace`, `owner_kind`, `owner_name` and `node`, plus the `severity`,
`exported_instance` and `service` labels the other platform alerts set,
and its value is the number of pods on that node. DaemonSets, Jobs and
every other owner kind are not in the list, so a DaemonSet surge pod
next to its predecessor does not count.

etcd-operator gives every member pod its own `EtcdMember` as controller
owner
([etcdmember_controller.go#L546](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L546)),
so grouping by owner would see one pod per group. Members get their
names from `GenerateName: <cluster>-`
([etcdcluster_controller.go#L434](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L434),
[#L839](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L839)),
and members adopted from the old StatefulSet keep names like `etcd-0`
([adopt.go#L103](https://github.com/cozystack/etcd-operator/blob/v0.5.6/internal/migrate/adopt.go#L103)).
The rule drops the last dash-separated part of the member name and
reports the group as `EtcdCluster <name>`. With the etcd package that is
one group per tenant namespace, and two etcd clusters in one namespace
would still be told apart. CNPG instance pods carry a controller
reference to their `Cluster`
([cluster_create.go#L1580](https://github.com/cloudnative-pg/cloudnative-pg/blob/v1.30.0/internal/controller/cluster_create.go#L1580)),
and Strimzi pods one to their `StrimziPodSet`
([StrimziPodSetController.java#L369](https://github.com/strimzi/strimzi-kafka-operator/blob/0.45.1-rc1/cluster-operator/src/main/java/io/strimzi/operator/cluster/operator/assembly/StrimziPodSetController.java#L369)).

Owner and node both come from `kube_pod_info`. kube-state-metrics fills
`created_by_kind` and `created_by_name` from the controller owner
reference and `node` from `spec.nodeName`
([pod.go#L638-L653](https://github.com/kubernetes/kube-state-metrics/blob/v2.15.0/internal/store/pod.go#L638-L653)).
`kube_pod_owner` and `kube_pod_status_phase` have no node label of their
own, and the kube-state-metrics scrape adds `node` from
`__meta_kubernetes_pod_node_name` with `honorLabels: true`
(`templates/kube-state-metrics-scrape.yaml`), so on those series `node`
is the node of the kube-state-metrics pod. The test data keeps that
label on the phase series. A pod counts while
`kube_pod_status_phase{phase="Running"}` is 1, so a Failed or Pending
pod on the same node does not. `kube_pod_labels` does carry every pod
label here (`--metric-labels-allowlist=pods=[*]`), but the rule does not
depend on it.

A node counts as schedulable unless it has a `NoSchedule` or `NoExecute`
taint. A cordon shows up as the
`node.kubernetes.io/unschedulable:NoSchedule` taint
([node_lifecycle_controller.go#L549-L555](https://github.com/kubernetes/kubernetes/blob/v1.33.12/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L549-L555)),
and not ready and unreachable nodes get theirs from their conditions
([#L87-L99](https://github.com/kubernetes/kubernetes/blob/v1.33.12/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L87-L99)),
so one clause covers all three, and a tainted control plane does not
count either. Pods that a drain had to put together on the remaining
nodes therefore stay quiet while the node is out, and the hour starts
once it is back.

The alert waits an hour. A replacement pod runs next to its terminating
predecessor only for the predecessor's grace period, and node
maintenance takes the drained node out of the count, while a placement
that has held for an hour does not fix itself. The alert is about a risk
rather than an outage, so the wait costs nothing. The VMAlert sets both
`remoteWrite` and `remoteRead`, so the pending state survives a vmalert
restart ([vmalert
docs](https://docs.victoriametrics.com/victoriametrics/vmalert/#alerts-state-on-restarts)).

Not covered are workloads that run one pod per owner, such as ClickHouse
with one StatefulSet per replica
([patterns.go#L45-L53](https://github.com/Altinity/clickhouse-operator/blob/release-0.25.2/pkg/model/chi/namer/patterns.go#L45-L53))
and the KubeVirt VMs behind a tenant Kubernetes node group. They would
need grouping by pod labels, which this rule avoids. The node count is
cluster-wide, so a workload that a node selector or affinity restricts
to fewer nodes than it has pods fires too.

`hack/workload-placement-alerts.bats` takes `.spec` of the VMRule and
replays it through `vmalert-tool` v1.136.0 pinned by digest, the release
vm-operator v0.68.4 deploys by default
([config.go#L38](https://github.com/VictoriaMetrics/operator/blob/v0.68.4/internal/config/config.go#L38))
and the same image #4963 and #4985 use.
`hack/testdata/workload-placement-alerts/unittest.yaml` has 13 cases
with 24 checks. Two of three StatefulSet pods on one node fire at 61
minutes and not at 59, with labels and annotations pinned. Spread pods,
five ReplicaSet pods on three nodes, a DaemonSet surge pod, two pods of
one Job, a Failed pod next to a running one, and one pod each in two
clusters that share workload and node names stay quiet. etcd members
fire as one `EtcdCluster` per namespace, for generated and adopted
member names. A replacement pod next to its terminating predecessor for
twenty minutes stays quiet, and the same overlap held past the hour
fires and resolves when it ends. A cordoned node keeps the alert quiet
until it is uncordoned, and a node with unreachable taints keeps it
quiet throughout. ReplicaSet, CNPG `Cluster` and `StrimziPodSet` pods
fire with their owner kind.

```console
$ bats hack/workload-placement-alerts.bats
1..1
ok 1 workload placement alerts pass their vmalert-tool cases
```

With the expression replaced by one that never fires, the suite prints
`not ok 1` and vmalert-tool reports 5 of the 24 checks, which are all
the checks that expect an alert. On main, where the rule file does not
exist, the suite fails as well. Each of these changes to the rule fails
at least one check, with the number of failing checks in brackets. `for:
15m` (5), no phase filter (1), no node count (6), no `cluster` label
(6), DaemonSet and Job in the kind list (3), no taint clause (4), `<`
instead of `<=` (5), etcd members grouped per member (1).

With Bats 1.14.0 at the commit CI pins, the suite passes together with
`alert-severity-contract`, `bats-strict-setup`,
`bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage`,
`bats-runner-contract`, `image-pin-consistency` and
`check-commit-trailers`, 109 of 109, and the severity contract lists the
new alert as `warning`. helm-unittest 1.1.2, the version CI pins, passes
the monitoring-agents chart with 19 of 19 tests in 6 suites, and `helm
template` with Helm 3.17.3 renders the VMRule the same as the file. Like
the migration suites the new suite needs docker, and only the image pull
touches the network. I have not run the rule in vmalert on a live
cluster.

On release-1.6 kube-state-metrics 2.15.0, its scrape config, the alerts
template, the VMAlert and vm-operator v0.68.4 are the same, and
etcd-operator v0.5.5 and CNPG 1.28.2 set the same owners and member
names. The new files apply there without changes. That backport is a
separate PR with one more commit that drops the `load test_helper` line,
because release-1.6 runs `hack/*.bats` through `hack/cozytest.sh`.

### Screenshots

Not a UI change.

### Downstream repositories

A new alert file under `packages/system/monitoring-agents/alerts/`, a
Bats suite under `hack/` and its test data. No values, schema, README,
image or `ApplicationDefinition` change, and the website does not list
platform alerts by name, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(monitoring): new warning alert PodReplicasColocated fires when two or more running pods of one StatefulSet, ReplicaSet, CNPG Cluster, StrimziPodSet or etcd cluster have shared a node for an hour while the cluster has a schedulable node for each of them.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added monitoring for workloads whose running pod replicas are
colocated on the same node.
* Alerts include workload, node, cluster, and severity details, along
with quorum-loss guidance.
* Coverage includes StatefulSets, ReplicaSets, Clusters, StrimziPodSets,
and EtcdMembers.

* **Tests**
* Added validation for workload-placement alerts across scheduling,
availability, cluster-isolation, and pod-replacement scenarios.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 17:38:58 +03:00
2026-11-13 12:57:32 +00:00
2025-10-08 09:43:34 +05:00
2024-02-08 12:04:32 +01:00

Cozystack Cozystack

GitHub Release Apache-2.0 License Commit activity OpenSSF Best Practices OpenSSF Scorecard CLOMonitor Support

Cozystack

Cozystack is a free and open-source platform and framework for building clouds.

Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.

With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.

Use Cozystack to build your own cloud or provide a cost-effective development environment.

Cozystack user interface

Use-Cases

Documentation

The documentation is located on the cozystack.io website.

Read the Getting Started section for a quick start.

If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.

Versioning

Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.

Contributions

Contributions are highly appreciated and very welcomed!

In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.

You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.

If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.

Community

You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.

Project documents

Document What it covers
Governance How decisions are made and who makes them
Contributor Ladder How to grow from contributor to maintainer
Maintainers Who maintains the project
Code of Conduct What we expect of each other
Security Policy Reporting a vulnerability, and which releases get fixes
Adopters Who runs Cozystack in production
Roadmap Where the project is going

License

Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.

Commercial Support

A list of companies providing commercial support for this project can be found on official site.

Languages
Go 52%
Shell 34%
TypeScript 6.8%
Go Template 4.7%
Makefile 1.3%
Other 1.1%