Commit Graph
1 Commits
Author SHA1 Message Date
Yan Bondarenko 5632596fe8 feat(monitoring): alert when pods of one workload share a node
Two etcd members, two Redis Sentinels or both replicas of a storage
tier can end up on one node through a soft spread rule, a drain, or a
placement made before a spread rule existed. Nothing reported it, and
the scheduler does not move running pods, so it stays like that until
a pod is recreated. One node failure then costs the quorum or every
copy at once.

etcd-operator owns each member pod by its own EtcdMember, so members
are grouped by the EtcdCluster name their names start with. Owner and
node are read from kube_pod_info because kube_pod_owner and
kube_pod_status_phase get the node of the kube-state-metrics pod as a
target label. Nodes with a NoSchedule or NoExecute taint do not count,
which covers cordoned, not ready and unreachable nodes. The alert waits
an hour, longer than a replacement pod runs next to its terminating
predecessor or a node takes to drain and come back.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-20 13:38:55 +03:00