Files
cozystack/hack/workload-placement-alerts.bats
T
Yan Bondarenko 5632596fe8 feat(monitoring): alert when pods of one workload share a node
Two etcd members, two Redis Sentinels or both replicas of a storage
tier can end up on one node through a soft spread rule, a drain, or a
placement made before a spread rule existed. Nothing reported it, and
the scheduler does not move running pods, so it stays like that until
a pod is recreated. One node failure then costs the quorum or every
copy at once.

etcd-operator owns each member pod by its own EtcdMember, so members
are grouped by the EtcdCluster name their names start with. Owner and
node are read from kube_pod_info because kube_pod_owner and
kube_pod_status_phase get the node of the kube-state-metrics pod as a
target label. Nodes with a NoSchedule or NoExecute taint do not count,
which covers cordoned, not ready and unreachable nodes. The alert waits
an hour, longer than a replacement pod runs next to its terminating
predecessor or a node takes to drain and come back.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-20 13:38:55 +03:00

27 lines
1001 B
Bash

#!/usr/bin/env bats
load test_helper
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
VMALERT_TOOL_IMAGE="victoriametrics/vmalert-tool:v1.136.0@sha256:417346d7b6fdb66d8e3984887c2ad114535ec7d983eeef53381b56c776f333f1"
@test "workload placement alerts pass their vmalert-tool cases" {
docker info >/dev/null 2>&1 || {
echo "docker is required to run the cases in $VMALERT_TOOL_IMAGE" >&2
return 1
}
work=$(mktemp -d)
rc=0
yq '.spec' "$REPO_ROOT/packages/system/monitoring-agents/alerts/workload-placement.yaml" > "$work/rules.yaml" || rc=$?
cp "$REPO_ROOT/hack/testdata/workload-placement-alerts/unittest.yaml" "$work/unittest.yaml" || rc=$?
if [ "$rc" -eq 0 ]; then
docker run --rm --network none -v "$work:/w:ro" "$VMALERT_TOOL_IMAGE" \
unittest --files /w/unittest.yaml > "$work/out" 2>&1 || rc=$?
cat "$work/out"
fi
passed=$(grep -cx 'SUCCESS' "$work/out" 2>/dev/null || true)
rm -rf "$work"
[ "$rc" -eq 0 ]
[ "$passed" -eq 1 ]
}