mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
Two etcd members, two Redis Sentinels or both replicas of a storage tier can end up on one node through a soft spread rule, a drain, or a placement made before a spread rule existed. Nothing reported it, and the scheduler does not move running pods, so it stays like that until a pod is recreated. One node failure then costs the quorum or every copy at once. etcd-operator owns each member pod by its own EtcdMember, so members are grouped by the EtcdCluster name their names start with. Owner and node are read from kube_pod_info because kube_pod_owner and kube_pod_status_phase get the node of the kube-state-metrics pod as a target label. Nodes with a NoSchedule or NoExecute taint do not count, which covers cordoned, not ready and unreachable nodes. The alert waits an hour, longer than a replacement pod runs next to its terminating predecessor or a node takes to drain and come back. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
27 lines
1001 B
Bash
27 lines
1001 B
Bash
#!/usr/bin/env bats
|
|
|
|
load test_helper
|
|
|
|
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
|
|
VMALERT_TOOL_IMAGE="victoriametrics/vmalert-tool:v1.136.0@sha256:417346d7b6fdb66d8e3984887c2ad114535ec7d983eeef53381b56c776f333f1"
|
|
|
|
@test "workload placement alerts pass their vmalert-tool cases" {
|
|
docker info >/dev/null 2>&1 || {
|
|
echo "docker is required to run the cases in $VMALERT_TOOL_IMAGE" >&2
|
|
return 1
|
|
}
|
|
work=$(mktemp -d)
|
|
rc=0
|
|
yq '.spec' "$REPO_ROOT/packages/system/monitoring-agents/alerts/workload-placement.yaml" > "$work/rules.yaml" || rc=$?
|
|
cp "$REPO_ROOT/hack/testdata/workload-placement-alerts/unittest.yaml" "$work/unittest.yaml" || rc=$?
|
|
if [ "$rc" -eq 0 ]; then
|
|
docker run --rm --network none -v "$work:/w:ro" "$VMALERT_TOOL_IMAGE" \
|
|
unittest --files /w/unittest.yaml > "$work/out" 2>&1 || rc=$?
|
|
cat "$work/out"
|
|
fi
|
|
passed=$(grep -cx 'SUCCESS' "$work/out" 2>/dev/null || true)
|
|
rm -rf "$work"
|
|
[ "$rc" -eq 0 ]
|
|
[ "$passed" -eq 1 ]
|
|
}
|