OPSaid – 告警

Docs: Node Exporter

Mon, 24 Jan 2022 10:43:37 +0800

简要概述

appid = “oneops”

指标名	类型	说明
node_filesystem_avail_bytes	xx	分区还剩的空间
node_filesystem_size_bytes	xx	分区总的空间

公共标签

标签名	说明
api.opsaid.cn/fault-priority	P5
api.opsaid.cn/pm2-uuid	项目的UUID

全局默认告警规则

groups:
- name: default-global-host-2m
 interval: 2m
 rules:

检测周期较长，持续周期较长，用于全局判断

主机cpu使用率预警

alert: NodeCPUUseHigh60
expr: |
 (
 1 - (sum(increase(node_cpu_seconds_total{appid!~"uptime",mode="idle"}[1m])) by (appid,job,instance,serial_num) / sum(increase(node_cpu_seconds_total{appid!~"uptime"}[1m])) by (appid,job,instance,serial_num) )
 ) >= 0.6
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,CPU使用率大于60%(当前值:{{$value|humanizePercentage}})

主机iowait使用率预警

alert: NodeCPUIOWaitHigh20
expr: |
 (
 (sum(increase(node_cpu_seconds_total{appid!~"uptime",mode="iowait"}[1m])) by (appid,job,instance,serial_num) / sum(increase(node_cpu_seconds_total{appid!~"uptime"}[1m])) by (appid,job,instance,serial_num) )
 ) >= 0.2
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,iowait使用率大于20%(当前值:{{$value|humanizePercentage}})

主机最近1分钟负载预警

alert: NodeLoad1High30
expr: ceil(avg(node_load1{appid!~"uptime"}) by (appid,job,instance,serial_num)) > 30
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,load1大于30(当前值:{{$value}})

预测空间24小时内会满

基于前6小时数据集预测24小时内空间可能满，需持续1小时

alert: NodeFilesystemSpaceFillingUp
expr: |
 (
 node_filesystem_avail_bytes{appid="oneops",fstype!="",device="rootfs"} / node_filesystem_size_bytes{appid="oneops",fstype!="",device="rootfs"} * 100 < 40
 and
 predict_linear(node_filesystem_avail_bytes{appid="oneops",fstype!="",device="rootfs"}[6h], 24*60*60) < 0
 and
 node_filesystem_readonly{appid="oneops",fstype!="",device="rootfs"} == 0
 )
for: 1h
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 分区空间可能会在未来24小时内满

分区空间剩余可用率：(node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 预测空间可用率：predict_linear(node_filesystem_avail_bytes{appid=“oneops”,fstype!=""}[6h], 246060) 分区文件系统非只读：node_filesystem_readonly{appid=“oneops”,fstype!=""}

predict_linear(v range-vector, t scalar) 基于v指标的前6小时数据集合，预测当前时间至t秒后值可能为多少

磁盘空间可用率不足

alert: NodeFilesystemAlmostOutOfSpace
expr: |
 (
 node_filesystem_avail_bytes{appid="oneops",fstype!="",device="rootfs"} / node_filesystem_size_bytes{appid="oneops",fstype!="",device="rootfs"} * 100 < 5
 and
 node_filesystem_readonly{appid="oneops",fstype!="",device="rootfs"} == 0
 )
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,空间可用率小于5%(当前值:{{$value|humanizePercentage}})

预测inode24小时内会满

alert: NodeFilesystemFilesFillingUp
expr: |
 (
 node_filesystem_files_free{appid="oneops",fstype!="",device="rootfs"} / node_filesystem_files{appid="oneops",fstype!="",device="rootfs"} * 100 < 40
 and
 predict_linear(node_filesystem_files_free{appid="oneops",fstype!="",device="rootfs"}[6h], 24*60*60) < 0
 and
 node_filesystem_readonly{appid="oneops",fstype!="",device="rootfs"} == 0
 )
for: 1h
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 分区inode可能会在未来24小时内满

分区inode可用率不足

alert: NodeFilesystemAlmostOutOfSpace
expr: |
 (
 node_filesystem_files_free{appid="oneops",fstype!="",device="rootfs"} / node_filesystem_files{appid="oneops",fstype!="",device="rootfs"} * 100 < 5
 and
 node_filesystem_readonly{appid="oneops",fstype!="",device="rootfs"} == 0
 )
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,inode可用率小于5%(当前值:{{$value|humanizePercentage}})

主机网卡数据包接收错误

alert: NodeNetworkReceiveErrs
expr: |
 rate(node_network_receive_errs_total{appid="oneops"}[2m]) / rate(node_network_receive_packets_total{appid="oneops"}[2m]) > 0.01
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,网卡{{ $labels.device }}存在接收数据包接收错误

conntrack使用条目是否快要达到限制

alert: NodeHighNumberConntrackEntriesUsed
expr: |
 (node_nf_conntrack_entries{appid!~"uptime"} / node_nf_conntrack_entries_limit{appid!~"uptime"}) > 0.75
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,conntrack数量使用率快要达到上限(当前值:{{$value|humanizePercentage}})

机器时钟发生漂移

alert: NodeClockSkewDetected
expr: |
 (
 node_timex_offset_seconds{appid!~"uptime"} > 0.05
 and
 deriv(node_timex_offset_seconds{appid!~"uptime"}[5m]) >= 0
 )
 or
 (
 node_timex_offset_seconds{appid!~"uptime"} < -0.05
 and
 deriv(node_timex_offset_seconds{appid!~"uptime"}[5m]) <= 0
 )
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,机器时间漂移超过300秒

机器时间未同步

alert: NodeClockNotSynchronising
expr: |
 min_over_time(node_timex_sync_status{appid!~"uptime"}[5m]) == 0
 and
 node_timex_maxerror_seconds{appid!~"uptime"} >= 16
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,机器时间没有在同步

文件描述符检测

告警规则

alert: NodeFileDescriptorLimit
expr: (node_filefd_allocated{appid!~"uptime"}/node_filefd_maximum{appid!~"uptime"}) * 100 > 70
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,文件描述符使用率快要达到上限(当前值:{{$value|humanizePercentage}})

规则解析

/proc/sys/fs/file-max

这个文件决定了当前内核可以打开的文件描述符的数量限制

/proc/sys/fs/file-nr

这个是一个状态指示的文件，一共三个值，第一个代表全局已经分配的文件描述符数量，第二个代表自由的文件描述符（待重新分配的），第三个代表总的文件描述符的数量。

Swap使用达到50%预警

告警规则

alert: NodeMemorySwapUsedHigh50
expr: ((node_memory_SwapTotal_bytes{appid!~"uptime"} - node_memory_SwapFree_bytes{appid!~"uptime"}) / (node_memory_SwapTotal_bytes{appid!~"uptime"})) > 0.5
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,主机swap使用达到50%以上(当前值:{{$value|humanizePercentage}})

规则解析

pass

进程已打开fd达到最大50%预警

alert: NodeFileDescriptorProcessesUsedHigh30
expr: (process_open_fds{appid!~"uptime"}>process_max_fds{appid!~"uptime"}) > 0.3
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,主机进程已打开fd达到最大50%预警当前值:{{$value|humanizePercentage}})

规则解析

依赖客户端上报数据

网卡数量过多

- alert: NodeNetworkCardCountHigh600
 expr: sort_desc(count(node_network_up{}) by(job,instance,host_ip)) > 600
 for: 6m
 labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
 annotations:
 summary: 2分钟检测一次,持续6分钟,主机数量大于600预警,当前值:{{$value}})

如果网卡数量过多，会导致采集的 node_exporter 进程卡死，一般一个网卡需要 3k 字节（未压缩）。

网卡接收数据带宽利用率达到90%

告警规则

- alert: NodeNetworkReceiveUsedHigh90
 expr: |
 (
 (
 max by(appid,job,serial_num,device,host_ip,instance) (rate(node_network_receive_bytes_total{appid!~"uptime"}[2m])*8)
 ) / 1024 / 1024
 )

 >

 (
 (
 max (node_network_speed_bytes{appid!~"uptime"} * 8 / 1000 / 1000) by (appid,job,serial_num,device,host_ip,instance) > 20
 ) * 0.9
 )
 for: 6m
 labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
 annotations:
 summary: 2分钟检测一次,持续6分钟,主机数量大于600预警,当前值:{{$value}})

规则解析

判断网卡速率，/sys/class/net/eth0/，对应指标 node_network_speed_bytes，它的单位为 “bytes” 转换 “bits” 需要乘以8。

sort_desc(node_network_speed_bytes{} * 8 / 1000 / 1000 > 10 )

过滤出大于 10Mbits 以上网卡

注意：

大部分虚机可能不存在该值

 - alert: NodeNetworkTransmitBytesTotalHigh1000Mbps
 expr: (max by(appid,job,serial_num,host_ip,instance)(rate(node_network_transmit_bytes_total{}[2m])*8)) / 1024 / 1024 > 1000
 for: 6m
 labels:
 ops_internal_17173_com_fault_priority: P5
 ops_internal_17173_com_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
 annotations:
 summary: 2分钟检测一次,持续6分钟,出流量大于1000Mbps(当前值:{{$value}}Mbps)

Docs: Etcd Exporter

Mon, 24 Jan 2022 10:43:37 +0800

简要概述

对来自 etcd 采集的性能数据做关键告警。

全局默认告警

集群成员在线数量异常

alert: EtcdMembersCountInvaild
expr: count by (appid) (up{appid!~"uptime",job=~"etcd.*exporter"}) < 3
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,etcd集群数量基数小于3(当前值:{{$value}})

集群是否存在leader节点

alert: EtcdNoLeader
expr: (etcd_server_has_leader{appid!~"uptime",job=~"etcd.*exporter"}) == 0
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,etcd集群未发现leader角色

集群leader切换频繁

alert: EtcdHighNumberOfLeaderChanges
expr: ceil(increase(etcd_server_leader_changes_seen_total{appid!~"uptime",job=~"etcd.*exporter"}[10m])) >= 3
for: 0m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 10分钟内,etcd集群leader切换大于3次(当前值:{{$value}})

集群间同步延迟高

alert: EtcdMemberCommunicationSlow
expr: histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{appid!~"uptime"}[5m])) >= 0.3
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,etcd集群99%的成员间同步延迟高于0.3秒(当前值:{{$value}})

集群过去1小时proposal请求失败

提议是一个需要完成 raft 协议的请求

alert: EtcdHighNumberOfFailedProposals
expr: ceil(increase(etcd_server_proposals_failed_total{appid!~"uptime"}[1h])) >= 10
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: etcd集群过去1小时内,proposal请求大于10(当前值:{{$value}})

集群wal fsync延迟高

alert: EtcdHighFsyncDurations
expr: histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{appid!~"uptime"}[2m])) >= 1
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,etcd集群99%的wal fsync延迟高于1秒(当前值:{{$value}})

集群commit延迟高

alert: EtcdHighFsyncDurations
expr: histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{appid!~"uptime"}[2m])) >= 1
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,etcd集群99%的commit延迟高于1秒(当前值:{{$value}})

集群空间快达到配额

alert: EtcdBackendQuotaLowSpace
expr: (etcd_mvcc_db_total_size_in_bytes{appid!~"uptime"}/etcd_server_quota_backend_bytes{appid!~"uptime"}) * 100 > 60
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,etcd集群空间使用率已达配额60%(当前值:{{$value}})

Docs: Kube State Metrics

Wed, 16 Feb 2022 16:40:10 +0800

简要概述

kube-state-metrics

K8S应用监控

Pod出现CrashLoopBackOff状态

告警规则

alert: KubePodStatusCrashLooping
expr: |
 max_over_time(kube_pod_container_status_waiting_reason{appid!~"uptime",reason="CrashLoopBackOff"}[5m]) >= 1
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,k8s集群中pod存在CrashLoopBackOff状态

原理分析

pass

Pod部署拉取镜像时间过长

告警规则

alert: KubePodStatusErrImagePull

alert: KubePodStatusImagePullBackOff
expr: |
 max_over_time(kube_pod_container_status_waiting_reason{appid!~"uptime",reason="ImagePullBackOff"}[5m]) >= 1
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,k8s集群中pod部署镜像未成功拉取

原理分析

pass

Pod出现非Running或Succeeded状态

alert: KubePodStatusNotRunning
expr: kube_pod_status_phase{appid!~"uptime",phase!~"Running|Succeeded"} == 1
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,k8s集群中pod存在非Running状态

Deployment部署Generation与期望不一致

告警规则

alert: KubeDeploymentGenerationMismatch
expr: |
 kube_deployment_status_observed_generation{appid!~"uptime"} != kube_deployment_metadata_generation{appid!~"uptime"}
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,k8s集群中deployment部署generation与期望不一致

规则解析

每次变更deployment资源，metadata.generation值会加1，之后等待控制器同步，以便使status.observedGeneration值等于metadata.generation，所以这种情况：observedGeneration 只能是小于等于 generation，如果持续长时间不等，就需要介入排查，相关源码主要在以下路径：

pkg/controller/deployment/sync.go

通过代码中，还有一种场景为：observedGeneration 大于等于 generation，目前这还不清楚具体情况，待测试(TODO)；

Deployment管理的Pod副本与期望不等

告警规则

alert: KubeDeploymentReplicasMismatch
expr: |
 (
 kube_deployment_spec_replicas{appid!~"uptime"} != kube_deployment_status_replicas_available{appid!~"uptime"}
 )
 and
 (
 changes(kube_deployment_status_replicas_updated{appid!~"uptime"}[10m]) == 0
 )
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,k8s集群中deployment存在副本数与期望不一致

规则解析

排除近10分钟在滚动更新的deployment资源，判断deployment期望与当前实际运行的副本是否一致。

( changes(kube_deployment_status_replicas_updated{appid!~“uptime”}[10m]) == 0) 用于计算最近10分钟内deployment.status.updatedReplicas 没有发生变化，这个一般在部署时会出现数量变更。

预警处理

TODO;

StatefulSet管理的Pod副本与期望不等

告警规则

alert: KubeStatefulSetReplicasMismatch
expr: |
 (
 kube_statefulset_spec_replicas{appid!~"uptime"} != kube_statefulset_status_replicas_available{appid!~"uptime"}
 )
 and
 (
 changes(kube_statefulset_status_replicas_updated{appid!~"uptime"}[10m]) == 0
 )
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,k8s集群中deployment存在副本数与期望不一致

规则解析

排除近10分钟在滚动更新的statefulset资源，判断statefulset期望与当前实际运行的副本是否一致。

( changes(kube_statefulset_status_replicas_updated{appid!~“uptime”}[10m]) == 0) 用于计算最近10分钟内statefulset.status.updatedReplicas 没有发生变化，这个一般在部署时会出现数量变更。

预警处理

TODO;

StatefulSet部署Generation与期望不一致

告警规则

alert: KubeStatefulSetGenerationMismatch
expr: |
 kube_statefulset_status_observed_generation{appid!~"uptime"} != kube_statefulset_metadata_generation{appid!~"uptime"}
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,k8s集群中statefulset部署generation与期望不一致

规则解析

每次变更statefulset资源，metadata.generation值会加1，之后等待控制器同步，以便使status.observedGeneration值等于metadata.generation，所以这种情况：observedGeneration 只能是小于等于 generation，如果持续长时间不等，就需要介入排查，相关源码主要在以下路径：

pkg/controller/statefulset/stateful_set_control.go

通过代码中，还有一种场景为：observedGeneration 大于等于 generation，目前这还不清楚具体情况，待测试(TODO)；

StatefulSet更新未成功

告警规则

alert: KubeStatefulSetUpdateNotRolledOut
expr: |
 (
 max without (revision) (
 kube_statefulset_status_current_revision{appid!~"uptime"}
 unless
 kube_statefulset_status_update_revision{appid!~"uptime"}
 )
 *
 (
 kube_statefulset_replicas{appid!~"uptime"}
 !=
 kube_statefulset_status_replicas_updated{appid!~"uptime"}
 )
 ) and (
 changes(kube_statefulset_status_replicas_updated{appid!~"uptime"}[5m])
 ==
 0
 )
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,k8s集群中statefulset更新存在失败

规则解析

TODO

DaemonSet更新未成功

告警规则

alert: KubeDaemonSetRolloutStuck
expr: |
 (
 (
 kube_daemonset_status_current_number_scheduled{appid!~"uptime"}
 !=
 kube_daemonset_status_desired_number_scheduled{appid!~"uptime"}
 ) or (
 kube_daemonset_status_number_misscheduled{appid!~"uptime"}
 !=
 0
 ) or (
 kube_daemonset_status_updated_number_scheduled{appid!~"uptime"}
 !=
 kube_daemonset_status_desired_number_scheduled{appid!~"uptime"}
 ) or (
 kube_daemonset_status_number_available{appid!~"uptime"}
 !=
 kube_daemonset_status_desired_number_scheduled{appid!~"uptime"}
 )
 ) and (
 changes(kube_daemonset_status_updated_number_scheduled{appid!~"uptime"}[5m])
 ==
 0
 )
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,k8s集群中daemonset更新存在失败

规则解析

TODO

DaemonSet存在未成功调度

告警规则

alert: KubeDaemonSetNotScheduled
expr: (kube_daemonset_status_desired_number_scheduled{appid!~"uptime"} - kube_daemonset_status_current_number_scheduled{appid!~"uptime"}) > 0
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,k8s集群中daemonset存在未成功调度节点

规则解析

集群中期望被调度的节点数，减去当前实际已经调度的节点数，如果大于0则说明集群中存在未成功调度。

Pod等待过久

告警规则

alert: KubeContainerWaiting
expr: sum by (appid,namespace,pod,container,reason) (kube_pod_container_status_waiting_reason{appid!~"uptime"}) > 0
for: 30m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续30分钟,k8s集群中存在容器还处于等待启动中

规则解析

通过定期获取 pod.status.containerStatuses 数组，循环判断 item.state.waiting 是否非空，如果存在则记录 item.state.waiting.reason 值。

github.com/kubernetes/kube-state-metrics/internal/store/pod.go

func createPodContainerStatusWaitingReasonFamilyGenerator() generator.FamilyGenerator {
 return *generator.NewFamilyGenerator(
 "kube_pod_container_status_waiting_reason",
 "Describes the reason the container is currently in waiting state.",
 metric.Gauge,
 "",
 wrapPodFunc(func(p *v1.Pod) *metric.Family {
 ms := make([]*metric.Metric, 0, len(p.Status.ContainerStatuses))
 for _, cs := range p.Status.ContainerStatuses {
 // Skip creating series for running containers.
 if cs.State.Waiting != nil {
 ms = append(ms, &metric.Metric{
 LabelKeys: []string{"container", "reason"},
 LabelValues: []string{cs.Name, cs.State.Waiting.Reason},
 Value: 1,
 })
 }
 }
 return &metric.Family{
 Metrics: ms,
 }
 }),
 )
}

DaemonSet在不该被调度的节点上运行

告警规则

alert: KubeDaemonSetMisScheduled
expr: kube_daemonset_status_number_misscheduled{appid!~"uptime"} > 0
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,k8s集群中daemonset存在不匹配的调度

规则解析

daemonset在集群中应该有按照特定的标签调度部署至不同节点，可能存在某些场景导致pod在非期望的节点上运行，可以通过定期获取 daemonset.status.numberMisscheduled值，以判断是否有不匹配的调度。

github.com/kubernetes/kubernetes-1.18.8/pkg/controller/daemon/daemon_controller.go

Job执行时间过长

告警规则

alert: KubeJobNotCompleted
expr: |
 (
 time() - max by(namespace, job_name) (kube_job_status_start_time{appid!~"uptime"}
 and
 kube_job_status_active{appid!~"uptime"} > 0) > 43200
 )
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,k8s集群中Job {{ $labels.namespace }}/{{ $labels.job_name }} 执行时间超过 {{ "43200" | humanizeDuration }}

规则解析

获取当前正在运行的job列表，由当前时间减去任务开始执行时间，获取时差是否超过指定阈值。

Job执行失败

告警规则

alert: KubeJobFailed
expr: kube_job_failed{appid!~"uptime"} > 0
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,k8s集群中Job {{ $labels.namespace }}/{{ $labels.job_name }} 执行失败

规则解析

获取 job.status.conditions 判断job是否执行失败，这个与 KubeJobNotCompleted 形成互补，有时候任务一直无法结束，这种情况下就不存在conditions数组值。

K8S资源监控

CPU超过

告警规则

alert: KubeCPUOvercommit
expr: kube_job_failed{appid!~"uptime"} > 0
for: 6m
labels:
 api_opsaid_cn_fault_priority: P5
 api_opsaid_cn_pm2_uuid: 99feafb5-bed6-4daf-927a-69a2ab80c485
annotations:
 summary: 2分钟检测一次,持续6分钟,k8s集群中Job {{ $labels.namespace }}/{{ $labels.job_name }} 执行失败

规则解析

kube_pod_container_resource_requests{resource="cpu",namespace="user-cy7053",container="cms3html"}