Prometheusalertmanager邮件发送+grafana告警展示

前面一篇博客，我已经介绍了prometheus如何监控mysql。

这一篇我来介绍如何通过alertmanger进行告警邮件发送（微信或钉钉类似，因为需要企业帐户，我就不试了），以及如何通过grafana查看告警。

测试机器

Prometheus： 192.168.56.140

Host01：192.168.56.103

安装alertmanager

获取安装包

wget https://github.com/prometheus/alertmanager/releases/download/v0.22.2/alertmanager-0.22.2.linux-amd64.tar.gz

创建目录

mkdir -p /etc/alertmanager/

mkdir -p /etc/alertmanager/data

mkdir -p /etc/alertmanager/template/

获取邮件模板

[root@prometheus-server template]# pwd

/etc/alertmanager/template

[root@prometheus-servertemplate]# wget https://raw.githubusercontent.com/prometheus/alertmanager/master/template/default.tmpl

复制文件到/etc/alertmanager目录

[root@prometheus-server ftpusr]cp ./alertmanager-0.22.2.linux-amd64/alertmanager* /etc/alertmanager/.

配置启动服务

[root@prometheus-server alertmanager]# cat /etc/systemd/system/alertmanager.service
[Unit]
Description=Alertmanager
After=network.target
[Service]
Type=simple
User=prometheus
ExecStart=/etc/alertmanager/alertmanager \
--config.file=/etc/alertmanager/alertmanager.yml \
--storage.path=/etc/alertmanager/data
Restart=on-failure

[Install]
WantedBy=multi-user.target

配置alertmanager邮件发送

如下我使用的是163邮箱来发送邮件。

如需使用SMTP服务，需要先开启服务。开启后，增加授权码，如下配置文件里面的smtp_auth_password填写的是授权码（而不是个人邮箱密码）

[root@prometheus-server alertmanager]# cat alertmanager.yml

global:
smtp_smarthost: 'smtp.163.com:25'
smtp_from: 'xxxx@163.com'
smtp_auth_username: 'xxxx@163.com'
smtp_auth_password: 'xxxxxxxxxxx'
smtp_require_tls: false

templates:
- '/etc/alertmanager/template/*.tmpl'

route:
group_by: ['alertname','cluster','service']
group_wait: 10s
group_interval: 10s
repeat_interval: 10m
receiver: 'default-receiver'

receivers:
- name: 'default-receiver'
email_configs:
- to: '20889922@qq.com'
html: '{{ template "email.default.html" . }}'
headers: { Subject: "Prometheus 告警测试邮件" }

启动服务

service alertmanager start

prometheus配置alertmanager

prometheus.yml配置

# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets: ["localhost:9093"]
# - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
- "rules.yml"
# - "first_rules.yml"
# - "second_rules.yml"

rules.yml配置

[root@prometheus-server prometheus]# cat rules.yml
# hostStatsAlert
groups:
- name: hostStatsAlert
rules:
- alert: NodeDown
expr: up == 0
for: 1m
labels:
severity: "Critical"
annotations:
summary: "Instance {{$labels.instance}} down"
description: "{{$labels.instance}} of job {{$labels.job}} has been down for more than 5 minutes."
- alert: NodeCPUUsage
expr: sum(avg without (cpu)(irate(node_cpu_seconds_total{mode!='idle'}[5m]))) by (instance) > 0.85
for: 1m
labels:
severity: "Warning"
annotations:
summary: "Instance {{ $labels.instance }} CPU usgae high"
description: "{{ $labels.instance }} CPU usage above 85% (current value: {{ $value }})"
- alert: NodeMemoryUsage
expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes)/node_memory_MemTotal_bytes > 0.85
for: 1m
labels:
severity: "Warning"
annotations:
summary: "Instance {{ $labels.instance }} MEM usgae high"
description: "{{ $labels.instance }} MEM usage above 85% (current value: {{ $value }})"
- alert: filesystemUsageAlert
expr: 100 - ((node_filesystem_avail_bytes{mountpoint="/",fstype=~"ext4|xfs"} * 100) / node_filesystem_size_bytes {mountpoint="/",fstype=~"ext4|xfs"}) > 85
for: 1m
labels:
severity: "Warning"
annotations:
summary: "Instance {{ $labels.instance }} root DISK usgae high"
description: "{{ $labels.instance }} root DISK usage above 85% (current value: {{ $value }})"

重新启动prometheus使服务生效

 service prometheus restart

查看告警邮件

等待几分钟后，可以看到邮件的告警信息

登录alertmanager端口，也可查看告警信息

http://192.168.56.140:9093/

Alertmanager grafana展示

安装

grafana-cli plugins install camptocamp-prometheus-alertmanager-datasource

安装完后，重新启动grafana-server

service grafana-server restart

添加alertmanager datasource

导入dashboard

展示效果

碰到的问题与解决方法

告警展示的时候，虽然alerts有两个告警，但downnode却显示没有。

通过下载展示的JSON文件，查看原来是altername在告警文件中，与JSON文件中不匹配。匹配完成就OK了。

serverity在邮件显示正常，但是grafana无法正常显示。这个还没调查清楚。

估计得需要谷歌了。但是，你能体会中国人无法上谷歌的痛苦吗？

https://www.cnblogs.com/danny-djy/p/11097726.html

https://medium.com/devops-dudes/prometheus-alerting-with-alertmanager-e1bbba8e6a8e