目录

一.总览

二.安装prometheus &＃43; grafana

三. 安装一个exporter

四.pushgateway安装

五. service discovery之consul

六.alertmanager

七.Grafana嵌入到自己的项目

八.Grafana API报告的导出为PDF

一.总览

prometheus提供的一张图

这张图包含几部分

1.监控grafana

2.pull metrics&＃xff1a;&＃xff08;1&＃xff09;pushgateway &＃xff08;2&＃xff09;exporters

3.服务发现service discovery&＃xff0c;主要是动态发现exporters

4.告警alertmanager

5.prometheus server

二.安装prometheus &＃43; grafana

1.在/ops/prometheus下建立文件prometheus.yml 和targets.json&＃xff08;这个可以不用&＃xff0c;也是用于动态添加targets&＃xff09;

编辑文件vim prometheus.ymlglobal:scrape_interval: 1sevaluation_interval: 1sscrape_configs:- job_name: prometheusstatic_configs:- targets: [&＃39;localhost:9090&＃39;]labels:instance: prometheus- job_name: pushgateway #安装的pushgateway使用该配置项metrics_path: &＃39;/metrics&＃39;static_configs:- targets: [&＃39;192.168.253.215:9091&＃39;] # 地址修改为实际地址labels:instance: pushgatewayhonor_labels: true- job_name: nodesfile_sd_configs:#提供文件方式动态添加targets- files:- &＃39;targets.json&＃39;refresh_interval: 10s编辑文件 targets.json &＃xff0c;node_exporter会用该配置项
[{"targets": ["192.168.253.215:9400"],"labels": {"job": "node"}}
]

2.启动prometheus

docker run  -d   -p 9090:9090   -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml    -v /opt/prometheus/targets.json:/etc/prometheus/targets.json    prom/prometheus

启动后访问地址 http://192.168.253.215:9090/

3.启动grafana

docker run -d \-p 3000:3000 \--name&＃61;grafana \-v /opt/grafana-storage:/var/lib/grafana \grafana/grafana

访问地址 

http://192.168.253.215:3000/

3.在grafana添加prometheus&＃xff0c;点击save保存就可以

4.grafana导入已经存在的图表配置

在这里下载一个模板locust &＃xff1a;https://grafana.com/grafana/dashboards/12081

导入

5.grafana刷新时间是最小5秒&＃xff0c;配置每秒刷新一次&＃xff0c;注&＃xff1a;网上说低版本有bug&＃xff0c;该配置项不起作用

进入docker 

docker exec  -it --user root  b1dbe0b4357a /bin/bash

修改配置&＃xff0c;修改后重启docker

vi /etc/grafana/grafana.ini搜min_refresh_interval改为1s
由&＃xff1a;
;min_refresh_interval &＃61; 5s
改为&＃xff1a;
min_refresh_interval &＃61; 1s

6.打开grafana相关的工程&＃xff0c;配置刷新时间为1s

非docker安装

grafana下载包&＃xff1a;https://grafana.com/grafana/download?platform&＃61;linux
./grafana-server -config&＃61;/home/ops/grafana-7.3.5/conf/grafana.ini cfg:default.paths.plugins&＃61;/home/ops/grafana-7.3.5/plugins
 

三. 安装一个exporter

1.下载地址 https://github.com/prometheus/node_exporter/releases

2.启动 ./node_exporter --web.listen-address&＃61;":9400"

3. 前面prometheus已经配置了9400这个ip地址的targets&＃xff0c;所有现在应该已经开始监控

4. 随便打开一个面板图&＃xff0c;在面板最下面添加一个【Query】&＃xff0c;选择【Metrics】&＃61;》【node】&＃61;》【node_load1】,输入框可以根据条件过滤 node_load1{instance&＃61;"192.168.253.215:9400", job&＃61;"node"}

5.修改面板的刷新时间是1S

四.pushgateway安装

拓扑图

1. docker 安装 docker run -d -p 9091:9091 prom/pushgateway

2.启动后可以访问 http://192.168.253.215:9091/

3.推一条指标给pushgateway

cat <locust_stats_avg_content_length 2398.283
EOF

4.在上面prometheus已经配置好pushgateway&＃xff0c;如果没配&＃xff0c;先配置Prometheus

5.删除指标&＃xff0c;也可以去界面删除  curl -X DELETE http://192.168.253.215:9091/metrics/job/some_job/instance/some_instance

6.python​用 prometheus_client实现参考链接&＃xff1a;

https://www.cnblogs.com/xiao987334176/p/9933963.html

五. service discovery之consul

1.安装consul

docker run --name consul -d -p 8500:8500 consul

2.配置prometheus

- job_name: &＃39;consul&＃39;consul_sd_configs:- server: &＃39;192.168.253.215:8500&＃39;services: []relabel_configs:- source_labels: [__meta_consul_tags]regex: .*dev.*action: keep

3.向consul注册服务

curl -X PUT -d &＃39;{"id": "test1","name": "test1","address": "192.168.253.215","port": 9500,"tags": ["dev"],"checks": [{"http": "http://192.168.253.215:9500/","interval": "5s"}]}&＃39; http://192.168.253.215:8500/v1/agent/service/register注销服务
curl -X PUT http://192.168.253.215:8500/v1/agent/service/deregister/服务id


4.启动服务node_exporter :

./node_exporter --web.listen-address&＃61;":9500"

5.查看注册成功 &＃xff1a;http://192.168.253.215:8500/ui/dc1/services/test1/instances

6.查看peometheus是否正常 http://192.168.253.215:9090/targets

7.查看grafana&＃xff0c;多了一条曲线

六.alertmanager

七.Grafana嵌入到自己的项目

八.Grafana API报告的导出为PDF

由于Grafana插件不支持docker&＃xff0c;需要自己安装grafana程序

wget https://dl.grafana.com/oss/release/grafana-7.3.5-1.x86_64.rpm
sudo yum install grafana-7.3.5-1.x86_64.rpm
service grafana-server startyum install -y libXcomposite libXdamage libXtst cups libXScrnSaver pango atk adwaita-cursor-theme adwaita-icon-theme at at-spi2-atk at-spi2-core cairo-gobject colord-libs dconf desktop-file-utils ed emacs-filesystem gdk-pixbuf2 glib-networking gnutls gsettings-desktop-schemas gtk-update-icon-cache gtk3 hicolor-icon-theme jasper-libs json-glib libappindicator-gtk3 libdbusmenu libdbusmenu-gtk3 libepoxy liberation-fonts liberation-narrow-fonts liberation-sans-fonts liberation-serif-fonts libgusb libindicator-gtk3 libmodman libproxy libsoup libwayland-cursor libwayland-egl libxkbcommon m4 mailx nettle patch psmisc redhat-lsb-core redhat-lsb-submod-security rest spax time trousers xdg-utils xkeyboard-config

安装插件和grafana-reporter

grafana-cli plugins install grafana-image-renderer安装一下库&＃xff0c;不然grafana-reporter会报错
yum install texlive

在第七步的基础上配置Grafana

手动下载pdf&＃xff0c;grafana的看板是中文时会有问题&＃xff0c;但指标是中文&＃xff0c;没问题

九.prometheus API的使用

获取到所有job
http://192.168.253.215:9090/api/v1/label/job/values查询10秒内数据
http://192.168.253.215:9090/api/v1/query_range?query&＃61;network_traffic_input&start&＃61;1607999428.447&end&＃61;1607999468.447&step&＃61;10s查询指标
http://192.168.253.215:9090/api/v1/series?match[]&＃61;up{job&＃61;\"network_traffic\"}&match[]&＃61;network_traffic_input{adapter_name&＃61;\"ens33\"}查询当前
http://192.168.253.215:9090/api/v1/query?query&＃61;network_traffic_input查询
http://192.168.253.215:9090/api/v1/query_range?query&＃61;locust_stats_avg_content_length{instance&＃61;"some_instance1"} offset 15m&start&＃61;1608009390&end&＃61;1608012990&step&＃61;15
编码后&＃xff1a;
http://192.168.253.215:9090/api/v1/query_range?query&＃61;locust_stats_avg_content_length%7Binstance%3D%22some_instance1%22%7D%20offset%2015m&start&＃61;1608009390&end&＃61;1608012990&step&＃61;15consul注册服务
curl -X PUT -d &＃39;{"id": "test1","name": "test1","address": "192.24.17.156","port": 9500,"tags": ["dev"],"checks": [{"http": "http://192.24.17.156:9500/","interval": "5s"}]}&＃39;     http://192.24.17.156:8500/v1/agent/service/registergrafana下载包&＃xff1a;https://grafana.com/grafana/download?platform&＃61;linux
./grafana-server -config&＃61;/home/ops/grafana-7.3.5/conf/grafana.ini cfg:default.paths.plugins&＃61;/home/ops/grafana-7.3.5/plugins

参考文章&＃xff1a;

1.locust &＃43; Prometheus &＃43; grafana https://bugvanisher.cn/2020/04/05/locust-with-prometheus-and-grafana/

2.pushgateway安装使用&＃xff1a; https://www.cnblogs.com/xiaobaozi-95/p/10684524.html

3.prometheus file_sd_config配置项 https://www.cnblogs.com/faberbeta/p/13597816.html

4.prometheus consul&＃xff1a;https://blog.51cto.com/1000682/2363038

5.alertmanager&＃xff1a;https://www.cnblogs.com/xiaobaozi-95/p/10740511.html

6.动态更新prometheus 报警规则 https://www.lijiaocn.com/%E6%8A%80%E5%B7%A7/2018/08/30/confd-prometheus-dynamic-config.html

7.Grafana嵌入到自己的项目&＃xff1a;https://blog.csdn.net/weixin_44534391/article/details/107019567?utm_medium&＃61;distribute.pc_relevant.none-task-blog-baidulandingword-3&spm&＃61;1001.2101.3001.4242

8.Grafana导出pdf &＃xff1a;https://blog.csdn.net/shen12138/article/details/110199638

9.pushgateway python 使用https://www.cnblogs.com/xiao987334176/p/9933963.html