mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 22:13:18 +09:00
Yan Bondarenko
1ca7c0d891
The tenant etcd copy of etcdHighNumberOfFailedGRPCRequests counted every code other than OK. A Watch stream is counted once, when it ends, and the streams of a kube-apiserver that stops end with Canceled. So whenever an API server on that etcd goes away, the Watch method of every member reads as 100 percent failed and the critical rule fires on all of them. The rules now count the codes the etcd mixin counts since etcd-io/etcd#13127, the same list the platform copy in monitoring-agents already uses. The grouping by method stays, so the alert labels do not change. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
28 lines
1.1 KiB
Bash
28 lines
1.1 KiB
Bash
#!/usr/bin/env bats
|
|
|
|
load test_helper
|
|
|
|
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
|
|
VMALERT_TOOL_IMAGE="victoriametrics/vmalert-tool:v1.136.0@sha256:417346d7b6fdb66d8e3984887c2ad114535ec7d983eeef53381b56c776f333f1"
|
|
|
|
@test "tenant etcd failed gRPC request alerts pass their vmalert-tool cases" {
|
|
docker info >/dev/null 2>&1 || {
|
|
echo "docker is required to run the cases in $VMALERT_TOOL_IMAGE" >&2
|
|
return 1
|
|
}
|
|
work=$(mktemp -d)
|
|
helm template etcd "$REPO_ROOT/packages/extra/etcd" --namespace tenant-test \
|
|
--set '_cluster.cluster-domain=cozy.local' \
|
|
--show-only templates/prometheus-rules.yaml > "$work/rendered.yaml"
|
|
yq '.spec' "$work/rendered.yaml" > "$work/rules.yaml"
|
|
cp "$REPO_ROOT/hack/testdata/etcd-grpc-alerts/unittest.yaml" "$work/unittest.yaml"
|
|
rc=0
|
|
docker run --rm --network none -v "$work:/w:ro" "$VMALERT_TOOL_IMAGE" \
|
|
unittest --files /w/unittest.yaml > "$work/out" 2>&1 || rc=$?
|
|
cat "$work/out"
|
|
passed=$(grep -cx 'SUCCESS' "$work/out" || true)
|
|
rm -rf "$work"
|
|
[ "$rc" -eq 0 ]
|
|
[ "$passed" -eq 1 ]
|
|
}
|