mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
feat(monitoring): alert when pods of one workload share a node (#4993)
## What this PR does On three 3-node clusters running v1.6.4 and v1.6.5 we found a tenant etcd with two of its three members on one node, a tenant Redis with two of its three Sentinels on the node that also ran the master, and several VictoriaMetrics and VictoriaLogs pairs (vmstorage, vmselect, vmalertmanager, vlstorage) sharing a node. Nothing alerted. Losing one of those nodes would have cost the etcd or Sentinel quorum, or both copies of a storage tier. Spread constraints keep new pods apart once they are in place (#4986 for etcd, #4989 for monitoring, #4990 for Redis), but a placement that already exists, or that comes from a soft rule or a drain, stays as it is, because the scheduler does not move running pods. This adds `PodReplicasColocated`, a warning in a new `workload-placement` group under `packages/system/monitoring-agents/alerts/`. It fires when two or more running pods of one StatefulSet, ReplicaSet, CNPG `Cluster` or `StrimziPodSet` have been on one node for an hour while the cluster has at least as many schedulable nodes as the workload has running pods, so five replicas on three nodes stay quiet. The alert carries `cluster`, `namespace`, `owner_kind`, `owner_name` and `node`, plus the `severity`, `exported_instance` and `service` labels the other platform alerts set, and its value is the number of pods on that node. DaemonSets, Jobs and every other owner kind are not in the list, so a DaemonSet surge pod next to its predecessor does not count. etcd-operator gives every member pod its own `EtcdMember` as controller owner ([etcdmember_controller.go#L546](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L546)), so grouping by owner would see one pod per group. Members get their names from `GenerateName: <cluster>-` ([etcdcluster_controller.go#L434](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L434), [#L839](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L839)), and members adopted from the old StatefulSet keep names like `etcd-0` ([adopt.go#L103](https://github.com/cozystack/etcd-operator/blob/v0.5.6/internal/migrate/adopt.go#L103)). The rule drops the last dash-separated part of the member name and reports the group as `EtcdCluster <name>`. With the etcd package that is one group per tenant namespace, and two etcd clusters in one namespace would still be told apart. CNPG instance pods carry a controller reference to their `Cluster` ([cluster_create.go#L1580](https://github.com/cloudnative-pg/cloudnative-pg/blob/v1.30.0/internal/controller/cluster_create.go#L1580)), and Strimzi pods one to their `StrimziPodSet` ([StrimziPodSetController.java#L369](https://github.com/strimzi/strimzi-kafka-operator/blob/0.45.1-rc1/cluster-operator/src/main/java/io/strimzi/operator/cluster/operator/assembly/StrimziPodSetController.java#L369)). Owner and node both come from `kube_pod_info`. kube-state-metrics fills `created_by_kind` and `created_by_name` from the controller owner reference and `node` from `spec.nodeName` ([pod.go#L638-L653](https://github.com/kubernetes/kube-state-metrics/blob/v2.15.0/internal/store/pod.go#L638-L653)). `kube_pod_owner` and `kube_pod_status_phase` have no node label of their own, and the kube-state-metrics scrape adds `node` from `__meta_kubernetes_pod_node_name` with `honorLabels: true` (`templates/kube-state-metrics-scrape.yaml`), so on those series `node` is the node of the kube-state-metrics pod. The test data keeps that label on the phase series. A pod counts while `kube_pod_status_phase{phase="Running"}` is 1, so a Failed or Pending pod on the same node does not. `kube_pod_labels` does carry every pod label here (`--metric-labels-allowlist=pods=[*]`), but the rule does not depend on it. A node counts as schedulable unless it has a `NoSchedule` or `NoExecute` taint. A cordon shows up as the `node.kubernetes.io/unschedulable:NoSchedule` taint ([node_lifecycle_controller.go#L549-L555](https://github.com/kubernetes/kubernetes/blob/v1.33.12/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L549-L555)), and not ready and unreachable nodes get theirs from their conditions ([#L87-L99](https://github.com/kubernetes/kubernetes/blob/v1.33.12/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L87-L99)), so one clause covers all three, and a tainted control plane does not count either. Pods that a drain had to put together on the remaining nodes therefore stay quiet while the node is out, and the hour starts once it is back. The alert waits an hour. A replacement pod runs next to its terminating predecessor only for the predecessor's grace period, and node maintenance takes the drained node out of the count, while a placement that has held for an hour does not fix itself. The alert is about a risk rather than an outage, so the wait costs nothing. The VMAlert sets both `remoteWrite` and `remoteRead`, so the pending state survives a vmalert restart ([vmalert docs](https://docs.victoriametrics.com/victoriametrics/vmalert/#alerts-state-on-restarts)). Not covered are workloads that run one pod per owner, such as ClickHouse with one StatefulSet per replica ([patterns.go#L45-L53](https://github.com/Altinity/clickhouse-operator/blob/release-0.25.2/pkg/model/chi/namer/patterns.go#L45-L53)) and the KubeVirt VMs behind a tenant Kubernetes node group. They would need grouping by pod labels, which this rule avoids. The node count is cluster-wide, so a workload that a node selector or affinity restricts to fewer nodes than it has pods fires too. `hack/workload-placement-alerts.bats` takes `.spec` of the VMRule and replays it through `vmalert-tool` v1.136.0 pinned by digest, the release vm-operator v0.68.4 deploys by default ([config.go#L38](https://github.com/VictoriaMetrics/operator/blob/v0.68.4/internal/config/config.go#L38)) and the same image #4963 and #4985 use. `hack/testdata/workload-placement-alerts/unittest.yaml` has 13 cases with 24 checks. Two of three StatefulSet pods on one node fire at 61 minutes and not at 59, with labels and annotations pinned. Spread pods, five ReplicaSet pods on three nodes, a DaemonSet surge pod, two pods of one Job, a Failed pod next to a running one, and one pod each in two clusters that share workload and node names stay quiet. etcd members fire as one `EtcdCluster` per namespace, for generated and adopted member names. A replacement pod next to its terminating predecessor for twenty minutes stays quiet, and the same overlap held past the hour fires and resolves when it ends. A cordoned node keeps the alert quiet until it is uncordoned, and a node with unreachable taints keeps it quiet throughout. ReplicaSet, CNPG `Cluster` and `StrimziPodSet` pods fire with their owner kind. ```console $ bats hack/workload-placement-alerts.bats 1..1 ok 1 workload placement alerts pass their vmalert-tool cases ``` With the expression replaced by one that never fires, the suite prints `not ok 1` and vmalert-tool reports 5 of the 24 checks, which are all the checks that expect an alert. On main, where the rule file does not exist, the suite fails as well. Each of these changes to the rule fails at least one check, with the number of failing checks in brackets. `for: 15m` (5), no phase filter (1), no node count (6), no `cluster` label (6), DaemonSet and Job in the kind list (3), no taint clause (4), `<` instead of `<=` (5), etcd members grouped per member (1). With Bats 1.14.0 at the commit CI pins, the suite passes together with `alert-severity-contract`, `bats-strict-setup`, `bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage`, `bats-runner-contract`, `image-pin-consistency` and `check-commit-trailers`, 109 of 109, and the severity contract lists the new alert as `warning`. helm-unittest 1.1.2, the version CI pins, passes the monitoring-agents chart with 19 of 19 tests in 6 suites, and `helm template` with Helm 3.17.3 renders the VMRule the same as the file. Like the migration suites the new suite needs docker, and only the image pull touches the network. I have not run the rule in vmalert on a live cluster. On release-1.6 kube-state-metrics 2.15.0, its scrape config, the alerts template, the VMAlert and vm-operator v0.68.4 are the same, and etcd-operator v0.5.5 and CNPG 1.28.2 set the same owners and member names. The new files apply there without changes. That backport is a separate PR with one more commit that drops the `load test_helper` line, because release-1.6 runs `hack/*.bats` through `hack/cozytest.sh`. ### Screenshots Not a UI change. ### Downstream repositories A new alert file under `packages/system/monitoring-agents/alerts/`, a Bats suite under `hack/` and its test data. No values, schema, README, image or `ApplicationDefinition` change, and the website does not list platform alerts by name, so no trigger map entry applies. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(monitoring): new warning alert PodReplicasColocated fires when two or more running pods of one StatefulSet, ReplicaSet, CNPG Cluster, StrimziPodSet or etcd cluster have shared a node for an hour while the cluster has a schedulable node for each of them. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added monitoring for workloads whose running pod replicas are colocated on the same node. * Alerts include workload, node, cluster, and severity details, along with quorum-loss guidance. * Coverage includes StatefulSets, ReplicaSets, Clusters, StrimziPodSets, and EtcdMembers. * **Tests** * Added validation for workload-placement alerts across scheduling, availability, cluster-isolation, and pod-replacement scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
3 files changed
+679
No files matched your search
@@ -0,0 +1,572 @@
|
||||
rule_files:
|
||||
- rules.yaml
|
||||
|
||||
evaluation_interval: 15s
|
||||
|
||||
tests:
|
||||
- name: two of three StatefulSet pods on one node
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-0", node="node-a", created_by_kind="StatefulSet", created_by_name="db"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-1", node="node-a", created_by_kind="StatefulSet", created_by_name="db"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-2", node="node-b", created_by_kind="StatefulSet", created_by_name="db"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-0", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-1", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-2", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
alert_rule_test:
|
||||
- eval_time: 59m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
- eval_time: 61m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
cluster: cozystack
|
||||
namespace: tenant-a
|
||||
owner_kind: StatefulSet
|
||||
owner_name: db
|
||||
node: node-a
|
||||
severity: warning
|
||||
exported_instance: tenant-a/db
|
||||
service: workload-placement
|
||||
exp_annotations:
|
||||
summary: Pods of one workload share a node.
|
||||
description: >-
|
||||
2 running pods of StatefulSet tenant-a/db are on node node-a in cluster cozystack,
|
||||
although the cluster has a schedulable node for each of its pods. Losing this node
|
||||
stops these pods together, which can cost a quorum-based workload its quorum. The
|
||||
scheduler does not move running pods, so they stay together until they are recreated.
|
||||
|
||||
- name: StatefulSet pods spread over three nodes
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-0", node="node-a", created_by_kind="StatefulSet", created_by_name="db"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-1", node="node-b", created_by_kind="StatefulSet", created_by_name="db"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-2", node="node-c", created_by_kind="StatefulSet", created_by_name="db"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-0", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-1", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-2", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
alert_rule_test:
|
||||
- eval_time: 61m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
- eval_time: 120m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
|
||||
- name: five ReplicaSet pods on three nodes
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-aaaaa", node="node-a", created_by_kind="ReplicaSet", created_by_name="worker-7c8d9"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-bbbbb", node="node-a", created_by_kind="ReplicaSet", created_by_name="worker-7c8d9"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-ccccc", node="node-b", created_by_kind="ReplicaSet", created_by_name="worker-7c8d9"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-ddddd", node="node-b", created_by_kind="ReplicaSet", created_by_name="worker-7c8d9"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-eeeee", node="node-c", created_by_kind="ReplicaSet", created_by_name="worker-7c8d9"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-aaaaa", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-bbbbb", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-ccccc", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-ddddd", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-eeeee", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
alert_rule_test:
|
||||
- eval_time: 61m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
- eval_time: 120m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
|
||||
- name: DaemonSet surge pod next to the old one
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="cozy-system", pod="node-agent-x7k2p", node="node-a", created_by_kind="DaemonSet", created_by_name="node-agent"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="cozy-system", pod="node-agent-q9z4m", node="node-a", created_by_kind="DaemonSet", created_by_name="node-agent"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="cozy-system", pod="node-agent-b5n8w", node="node-b", created_by_kind="DaemonSet", created_by_name="node-agent"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="cozy-system", pod="node-agent-x7k2p", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="cozy-system", pod="node-agent-q9z4m", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="cozy-system", pod="node-agent-b5n8w", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
alert_rule_test:
|
||||
- eval_time: 61m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
- eval_time: 120m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
|
||||
- name: two pods of one Job on one node
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="backup-29345678-h2k4m", node="node-a", created_by_kind="Job", created_by_name="backup-29345678"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="backup-29345678-p8r6t", node="node-a", created_by_kind="Job", created_by_name="backup-29345678"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="backup-29345678-h2k4m", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="backup-29345678-p8r6t", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
alert_rule_test:
|
||||
- eval_time: 61m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
|
||||
- name: etcd members of one EtcdCluster on one node
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="etcd-x7k2p", node="node-a", created_by_kind="EtcdMember", created_by_name="etcd-x7k2p"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="etcd-q9z4m", node="node-a", created_by_kind="EtcdMember", created_by_name="etcd-q9z4m"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="etcd-b5n8w", node="node-b", created_by_kind="EtcdMember", created_by_name="etcd-b5n8w"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-b", pod="etcd-0", node="node-b", created_by_kind="EtcdMember", created_by_name="etcd-0"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-b", pod="etcd-1", node="node-c", created_by_kind="EtcdMember", created_by_name="etcd-1"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-b", pod="etcd-2", node="node-c", created_by_kind="EtcdMember", created_by_name="etcd-2"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="etcd-x7k2p", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="etcd-q9z4m", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="etcd-b5n8w", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-b", pod="etcd-0", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-b", pod="etcd-1", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-b", pod="etcd-2", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
alert_rule_test:
|
||||
- eval_time: 61m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
cluster: cozystack
|
||||
namespace: tenant-a
|
||||
owner_kind: EtcdCluster
|
||||
owner_name: etcd
|
||||
node: node-a
|
||||
severity: warning
|
||||
exported_instance: tenant-a/etcd
|
||||
service: workload-placement
|
||||
exp_annotations:
|
||||
summary: Pods of one workload share a node.
|
||||
description: >-
|
||||
2 running pods of EtcdCluster tenant-a/etcd are on node node-a in cluster cozystack,
|
||||
although the cluster has a schedulable node for each of its pods. Losing this node
|
||||
stops these pods together, which can cost a quorum-based workload its quorum. The
|
||||
scheduler does not move running pods, so they stay together until they are recreated.
|
||||
- exp_labels:
|
||||
cluster: cozystack
|
||||
namespace: tenant-b
|
||||
owner_kind: EtcdCluster
|
||||
owner_name: etcd
|
||||
node: node-c
|
||||
severity: warning
|
||||
exported_instance: tenant-b/etcd
|
||||
service: workload-placement
|
||||
exp_annotations:
|
||||
summary: Pods of one workload share a node.
|
||||
description: >-
|
||||
2 running pods of EtcdCluster tenant-b/etcd are on node node-c in cluster cozystack,
|
||||
although the cluster has a schedulable node for each of its pods. Losing this node
|
||||
stops these pods together, which can cost a quorum-based workload its quorum. The
|
||||
scheduler does not move running pods, so they stay together until they are recreated.
|
||||
|
||||
- name: replacement pod next to its terminating predecessor for twenty minutes
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-aaaaa", node="node-a", created_by_kind="ReplicaSet", created_by_name="web-6b8f4"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-bbbbb", node="node-b", created_by_kind="ReplicaSet", created_by_name="web-6b8f4"}'
|
||||
values: '1x30 stale'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-ccccc", node="node-b", created_by_kind="ReplicaSet", created_by_name="web-6b8f4"}'
|
||||
values: '_x10 1x140'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-aaaaa", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-bbbbb", phase="Running", node="node-c"}'
|
||||
values: '1x30 stale'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-ccccc", phase="Running", node="node-c"}'
|
||||
values: '_x10 1x140'
|
||||
alert_rule_test:
|
||||
- eval_time: 29m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
- eval_time: 75m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
- eval_time: 120m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
|
||||
- name: the same overlap held past the hour
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-aaaaa", node="node-a", created_by_kind="ReplicaSet", created_by_name="web-6b8f4"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-bbbbb", node="node-b", created_by_kind="ReplicaSet", created_by_name="web-6b8f4"}'
|
||||
values: '1x100 stale'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-ccccc", node="node-b", created_by_kind="ReplicaSet", created_by_name="web-6b8f4"}'
|
||||
values: '_x10 1x140'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-aaaaa", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-bbbbb", phase="Running", node="node-c"}'
|
||||
values: '1x100 stale'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-ccccc", phase="Running", node="node-c"}'
|
||||
values: '_x10 1x140'
|
||||
alert_rule_test:
|
||||
- eval_time: 69m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
- eval_time: 71m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
cluster: cozystack
|
||||
namespace: tenant-a
|
||||
owner_kind: ReplicaSet
|
||||
owner_name: web-6b8f4
|
||||
node: node-b
|
||||
severity: warning
|
||||
exported_instance: tenant-a/web-6b8f4
|
||||
service: workload-placement
|
||||
exp_annotations:
|
||||
summary: Pods of one workload share a node.
|
||||
description: >-
|
||||
2 running pods of ReplicaSet tenant-a/web-6b8f4 are on node node-b in cluster cozystack,
|
||||
although the cluster has a schedulable node for each of its pods. Losing this node
|
||||
stops these pods together, which can cost a quorum-based workload its quorum. The
|
||||
scheduler does not move running pods, so they stay together until they are recreated.
|
||||
- eval_time: 105m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
|
||||
- name: a cordoned node does not count until it is uncordoned
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_spec_unschedulable{job="kube-state-metrics", cluster="cozystack", node="node-a"}'
|
||||
values: '0x150'
|
||||
- series: 'kube_node_spec_unschedulable{job="kube-state-metrics", cluster="cozystack", node="node-b"}'
|
||||
values: '0x150'
|
||||
- series: 'kube_node_spec_unschedulable{job="kube-state-metrics", cluster="cozystack", node="node-c"}'
|
||||
values: '1x30 0x120'
|
||||
- series: 'kube_node_spec_taint{job="kube-state-metrics", cluster="cozystack", node="node-c", key="node.kubernetes.io/unschedulable", value="", effect="NoSchedule"}'
|
||||
values: '1x30 stale'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-0", node="node-a", created_by_kind="StatefulSet", created_by_name="kv"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-1", node="node-a", created_by_kind="StatefulSet", created_by_name="kv"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-2", node="node-b", created_by_kind="StatefulSet", created_by_name="kv"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-0", phase="Running", node="node-a"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-1", phase="Running", node="node-a"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-2", phase="Running", node="node-a"}'
|
||||
values: '1x150'
|
||||
alert_rule_test:
|
||||
- eval_time: 61m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
- eval_time: 90m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
- eval_time: 92m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
cluster: cozystack
|
||||
namespace: tenant-a
|
||||
owner_kind: StatefulSet
|
||||
owner_name: kv
|
||||
node: node-a
|
||||
severity: warning
|
||||
exported_instance: tenant-a/kv
|
||||
service: workload-placement
|
||||
exp_annotations:
|
||||
summary: Pods of one workload share a node.
|
||||
description: >-
|
||||
2 running pods of StatefulSet tenant-a/kv are on node node-a in cluster cozystack,
|
||||
although the cluster has a schedulable node for each of its pods. Losing this node
|
||||
stops these pods together, which can cost a quorum-based workload its quorum. The
|
||||
scheduler does not move running pods, so they stay together until they are recreated.
|
||||
|
||||
- name: a node that is down does not count
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_spec_unschedulable{job="kube-state-metrics", cluster="cozystack", node="node-a"}'
|
||||
values: '0x150'
|
||||
- series: 'kube_node_spec_unschedulable{job="kube-state-metrics", cluster="cozystack", node="node-b"}'
|
||||
values: '0x150'
|
||||
- series: 'kube_node_spec_unschedulable{job="kube-state-metrics", cluster="cozystack", node="node-c"}'
|
||||
values: '0x150'
|
||||
- series: 'kube_node_spec_taint{job="kube-state-metrics", cluster="cozystack", node="node-c", key="node.kubernetes.io/unreachable", value="", effect="NoSchedule"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_spec_taint{job="kube-state-metrics", cluster="cozystack", node="node-c", key="node.kubernetes.io/unreachable", value="", effect="NoExecute"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-0", node="node-a", created_by_kind="StatefulSet", created_by_name="kv"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-1", node="node-a", created_by_kind="StatefulSet", created_by_name="kv"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-2", node="node-b", created_by_kind="StatefulSet", created_by_name="kv"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-0", phase="Running", node="node-a"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-1", phase="Running", node="node-a"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-2", phase="Running", node="node-a"}'
|
||||
values: '1x150'
|
||||
alert_rule_test:
|
||||
- eval_time: 61m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
- eval_time: 120m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
|
||||
- name: pods that are not running do not count
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="api-5c6d7-aaaaa", node="node-a", created_by_kind="ReplicaSet", created_by_name="api-5c6d7"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="api-5c6d7-bbbbb", node="node-a", created_by_kind="ReplicaSet", created_by_name="api-5c6d7"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="api-5c6d7-ccccc", node="node-b", created_by_kind="ReplicaSet", created_by_name="api-5c6d7"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="api-5c6d7-aaaaa", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="api-5c6d7-bbbbb", phase="Running", node="node-c"}'
|
||||
values: '0x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="api-5c6d7-bbbbb", phase="Failed", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="api-5c6d7-ccccc", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
alert_rule_test:
|
||||
- eval_time: 61m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
|
||||
- name: one pod each in two clusters that share workload and node names
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="kubernetes-example", node="node-a"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="kubernetes-example", node="node-b"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="kubernetes-example", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="kube-system", pod="coredns-5d78c9869d-x7k2p", node="node-a", created_by_kind="ReplicaSet", created_by_name="coredns-5d78c9869d"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="kubernetes-example", namespace="kube-system", pod="coredns-5d78c9869d-q9z4m", node="node-a", created_by_kind="ReplicaSet", created_by_name="coredns-5d78c9869d"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="kube-system", pod="coredns-5d78c9869d-x7k2p", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="kubernetes-example", namespace="kube-system", pod="coredns-5d78c9869d-q9z4m", phase="Running", node="node-b"}'
|
||||
values: '1x150'
|
||||
alert_rule_test:
|
||||
- eval_time: 61m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
|
||||
- name: ReplicaSet, CNPG Cluster and StrimziPodSet pods on one node
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="rfs-cache-6f7d8b9c5d-h2k4m", node="node-a", created_by_kind="ReplicaSet", created_by_name="rfs-cache-6f7d8b9c5d"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="rfs-cache-6f7d8b9c5d-p8r6t", node="node-a", created_by_kind="ReplicaSet", created_by_name="rfs-cache-6f7d8b9c5d"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="rfs-cache-6f7d8b9c5d-w3x5z", node="node-b", created_by_kind="ReplicaSet", created_by_name="rfs-cache-6f7d8b9c5d"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="postgres-db-1", node="node-a", created_by_kind="Cluster", created_by_name="postgres-db"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="postgres-db-2", node="node-b", created_by_kind="Cluster", created_by_name="postgres-db"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="postgres-db-3", node="node-b", created_by_kind="Cluster", created_by_name="postgres-db"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kafka-kafka-0", node="node-c", created_by_kind="StrimziPodSet", created_by_name="kafka-kafka"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kafka-kafka-1", node="node-c", created_by_kind="StrimziPodSet", created_by_name="kafka-kafka"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kafka-kafka-2", node="node-a", created_by_kind="StrimziPodSet", created_by_name="kafka-kafka"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="rfs-cache-6f7d8b9c5d-h2k4m", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="rfs-cache-6f7d8b9c5d-p8r6t", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="rfs-cache-6f7d8b9c5d-w3x5z", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="postgres-db-1", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="postgres-db-2", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="postgres-db-3", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kafka-kafka-0", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kafka-kafka-1", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
- series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kafka-kafka-2", phase="Running", node="node-c"}'
|
||||
values: '1x150'
|
||||
alert_rule_test:
|
||||
- eval_time: 61m
|
||||
groupname: workload-placement
|
||||
alertname: PodReplicasColocated
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
cluster: cozystack
|
||||
namespace: tenant-a
|
||||
owner_kind: ReplicaSet
|
||||
owner_name: rfs-cache-6f7d8b9c5d
|
||||
node: node-a
|
||||
severity: warning
|
||||
exported_instance: tenant-a/rfs-cache-6f7d8b9c5d
|
||||
service: workload-placement
|
||||
exp_annotations:
|
||||
summary: Pods of one workload share a node.
|
||||
description: >-
|
||||
2 running pods of ReplicaSet tenant-a/rfs-cache-6f7d8b9c5d are on node node-a in cluster
|
||||
cozystack, although the cluster has a schedulable node for each of its pods. Losing this
|
||||
node stops these pods together, which can cost a quorum-based workload its quorum. The
|
||||
scheduler does not move running pods, so they stay together until they are recreated.
|
||||
- exp_labels:
|
||||
cluster: cozystack
|
||||
namespace: tenant-a
|
||||
owner_kind: Cluster
|
||||
owner_name: postgres-db
|
||||
node: node-b
|
||||
severity: warning
|
||||
exported_instance: tenant-a/postgres-db
|
||||
service: workload-placement
|
||||
exp_annotations:
|
||||
summary: Pods of one workload share a node.
|
||||
description: >-
|
||||
2 running pods of Cluster tenant-a/postgres-db are on node node-b in cluster cozystack,
|
||||
although the cluster has a schedulable node for each of its pods. Losing this node
|
||||
stops these pods together, which can cost a quorum-based workload its quorum. The
|
||||
scheduler does not move running pods, so they stay together until they are recreated.
|
||||
- exp_labels:
|
||||
cluster: cozystack
|
||||
namespace: tenant-a
|
||||
owner_kind: StrimziPodSet
|
||||
owner_name: kafka-kafka
|
||||
node: node-c
|
||||
severity: warning
|
||||
exported_instance: tenant-a/kafka-kafka
|
||||
service: workload-placement
|
||||
exp_annotations:
|
||||
summary: Pods of one workload share a node.
|
||||
description: >-
|
||||
2 running pods of StrimziPodSet tenant-a/kafka-kafka are on node node-c in cluster
|
||||
cozystack, although the cluster has a schedulable node for each of its pods. Losing
|
||||
this node stops these pods together, which can cost a quorum-based workload its quorum.
|
||||
The scheduler does not move running pods, so they stay together until they are
|
||||
recreated.
|
||||
@@ -0,0 +1,26 @@
|
||||
#!/usr/bin/env bats
|
||||
|
||||
load test_helper
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
|
||||
VMALERT_TOOL_IMAGE="victoriametrics/vmalert-tool:v1.136.0@sha256:417346d7b6fdb66d8e3984887c2ad114535ec7d983eeef53381b56c776f333f1"
|
||||
|
||||
@test "workload placement alerts pass their vmalert-tool cases" {
|
||||
docker info >/dev/null 2>&1 || {
|
||||
echo "docker is required to run the cases in $VMALERT_TOOL_IMAGE" >&2
|
||||
return 1
|
||||
}
|
||||
work=$(mktemp -d)
|
||||
rc=0
|
||||
yq '.spec' "$REPO_ROOT/packages/system/monitoring-agents/alerts/workload-placement.yaml" > "$work/rules.yaml" || rc=$?
|
||||
cp "$REPO_ROOT/hack/testdata/workload-placement-alerts/unittest.yaml" "$work/unittest.yaml" || rc=$?
|
||||
if [ "$rc" -eq 0 ]; then
|
||||
docker run --rm --network none -v "$work:/w:ro" "$VMALERT_TOOL_IMAGE" \
|
||||
unittest --files /w/unittest.yaml > "$work/out" 2>&1 || rc=$?
|
||||
cat "$work/out"
|
||||
fi
|
||||
passed=$(grep -cx 'SUCCESS' "$work/out" 2>/dev/null || true)
|
||||
rm -rf "$work"
|
||||
[ "$rc" -eq 0 ]
|
||||
[ "$passed" -eq 1 ]
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMRule
|
||||
metadata:
|
||||
name: alerts-workload-placement
|
||||
spec:
|
||||
groups:
|
||||
- name: workload-placement
|
||||
params: {}
|
||||
rules:
|
||||
- alert: PodReplicasColocated
|
||||
expr: |-
|
||||
count by (cluster, namespace, owner_kind, owner_name, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
(
|
||||
max by (cluster, namespace, pod, node, created_by_kind, created_by_name) (
|
||||
kube_pod_info{job="kube-state-metrics", created_by_kind=~"StatefulSet|ReplicaSet|Cluster|StrimziPodSet"}
|
||||
)
|
||||
or
|
||||
label_replace(
|
||||
label_replace(
|
||||
max by (cluster, namespace, pod, node, created_by_kind, created_by_name) (
|
||||
kube_pod_info{job="kube-state-metrics", created_by_kind="EtcdMember"}
|
||||
),
|
||||
"created_by_name", "$1", "created_by_name", "(.+)-[^-]+"
|
||||
),
|
||||
"created_by_kind", "EtcdCluster", "", ""
|
||||
)
|
||||
)
|
||||
and on (cluster, namespace, pod)
|
||||
kube_pod_status_phase{job="kube-state-metrics", phase="Running"} == 1,
|
||||
"owner_kind", "$1", "created_by_kind", "(.+)"
|
||||
),
|
||||
"owner_name", "$1", "created_by_name", "(.+)"
|
||||
)
|
||||
) > 1
|
||||
and on (cluster, namespace, owner_kind, owner_name)
|
||||
count by (cluster, namespace, owner_kind, owner_name) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
(
|
||||
max by (cluster, namespace, pod, node, created_by_kind, created_by_name) (
|
||||
kube_pod_info{job="kube-state-metrics", created_by_kind=~"StatefulSet|ReplicaSet|Cluster|StrimziPodSet"}
|
||||
)
|
||||
or
|
||||
label_replace(
|
||||
label_replace(
|
||||
max by (cluster, namespace, pod, node, created_by_kind, created_by_name) (
|
||||
kube_pod_info{job="kube-state-metrics", created_by_kind="EtcdMember"}
|
||||
),
|
||||
"created_by_name", "$1", "created_by_name", "(.+)-[^-]+"
|
||||
),
|
||||
"created_by_kind", "EtcdCluster", "", ""
|
||||
)
|
||||
)
|
||||
and on (cluster, namespace, pod)
|
||||
kube_pod_status_phase{job="kube-state-metrics", phase="Running"} == 1,
|
||||
"owner_kind", "$1", "created_by_kind", "(.+)"
|
||||
),
|
||||
"owner_name", "$1", "created_by_name", "(.+)"
|
||||
)
|
||||
)
|
||||
<= on (cluster) group_left ()
|
||||
count by (cluster) (
|
||||
max by (cluster, node) (kube_node_info{job="kube-state-metrics"})
|
||||
unless on (cluster, node)
|
||||
kube_node_spec_taint{job="kube-state-metrics", effect=~"NoSchedule|NoExecute"}
|
||||
)
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
exported_instance: '{{ $labels.namespace }}/{{ $labels.owner_name }}'
|
||||
service: workload-placement
|
||||
annotations:
|
||||
summary: Pods of one workload share a node.
|
||||
description: >-
|
||||
{{ $value }} running pods of {{ $labels.owner_kind }} {{ $labels.namespace }}/{{ $labels.owner_name }}
|
||||
are on node {{ $labels.node }} in cluster {{ $labels.cluster }}, although the cluster has a
|
||||
schedulable node for each of its pods. Losing this node stops these pods together, which can cost
|
||||
a quorum-based workload its quorum. The scheduler does not move running pods, so they stay
|
||||
together until they are recreated.
|
||||
Reference in new issue
Block a user