ELKStack实战篇（二)

说明

web01-node1.damaiche.org-203 logstash 收集数据
elk01-node2.damaiche.org-204 logstash(indexer) &＃43; kibana &＃43; es &＃43; redis

5.1 部署redis

机器:elk01-node2.damaiche.org-204

yum -y install redis

修改配置文件

vim /etc/redis.conf61 bind 10.0.0.204 127.0.0.1
128 daemonize yes

启动redis

systemctl restart redis

测试redis是否能正常写入

redis-cli -h 10.0.0.204 -p 637910.0.0.204:6379> set name hehe
OK
10.0.0.204:6379> get name
"hehe"

5.2 收集apache日志

5.2.1 收集日志到redis

机器&＃xff1a;web01-node1.damaiche.org-203

[root&＃64;web01-node1 conf.d]#cat redis.conf
input {
stdin{}file {path &＃61;> "/var/log/httpd/access_log"start_position &＃61;> "beginning"}
}output {redis {host &＃61;> [&＃39;10.0.0.204&＃39;]port &＃61;> &＃39;6379&＃39;db &＃61;> "6"key &＃61;> "apache_access_log"data_type &＃61;> "list"}
}

登录redis查看

10.0.0.204:6379> info # 查看redis详细的信息
......
# Keyspace
db0:keys&＃61;1,expires&＃61;0,avg_ttl&＃61;0
db6:keys&＃61;2,expires&＃61;0,avg_ttl&＃61;010.0.0.204:6379> select 6 # 选择key
OK10.0.0.204:6379[6]> keys * # 查看所有的key内容&＃xff0c;生产环境禁止操作
1) "apache_access_log"
2) "demo"10.0.0.204:6379[6]> type apache_access_log # 查看key的类型
list10.0.0.204:6379[6]> llen apache_access_log # 查看key的长度
(integer) 1010.0.0.204:6379[6]> lindex apache_access_log -1 # 查看最后一行的内容&＃xff0c;有得时候程序有问题&＃xff0c;想看最后一行的日志信息&＃xff0c;那么就可以这么查看。
"{\"message\":\"10.0.0.204 - - [04/May/2017:21:07:17 -0400] \\\"GET / HTTP/1.0\\\" 403 4897 \\\"-\\\" \\\"ApacheBench/2.3\\\"\",\"&＃64;version\":\"1\",\"&＃64;timestamp\":\"2017-05-05T01:07:17.422Z\",\"path\":\"/var/log/httpd/access_log\",\"host\":\"elk01-node2.damaiche.org-204\"}"
10.0.0.204:6379[6]> 启动到前台

5.2.2 启动logstash从redis读取数据

机器&＃xff1a;elk01-node2.damaiche.org-204

测试能否正常从redis读取数据&＃xff08;先不要写到es&＃xff09;

root&＃64;elk01-node2 conf.d]# cat indexer.conf
input {redis {host &＃61;> [&＃39;10.0.0204&＃39;]port &＃61;> &＃39;6379&＃39;db &＃61;> "6"key &＃61;> "apache_access_log"data_type &＃61;> "list"}
}filter {grok {match &＃61;> { "message" &＃61;> "%{COMBINEDAPACHELOG}"}}
}output{stdout {codec &＃61;> rubydebug}
}


造数据

[root&＃64;elk01-node2 conf.d]# ab -n 100 -c 1 http://10.0.0.203:8081/

此时到redis查看key access_apache_log的长度

10.0.0.204:6379[6]> llen apache_access_log
(integer) 100

将写好的indexer.conf启动到前台,此时再查看redis apache_access_log的长度

10.0.0.204:6379[6]> llen apache_access_log
(integer) 0长度为0,说明数据从redis里都取走了

此时将数据写入到es里

[root&＃64;elk01-node2 conf.d]# cat indexer.conf
input {redis {host &＃61;> [&＃39;10.0.0204&＃39;]port &＃61;> &＃39;6379&＃39;db &＃61;> "6"key &＃61;> "apache_access_log"data_type &＃61;> "list"}
}filter {# 如果收集多个日志文件&＃xff0c;那么这里一定要有type来进行判断&＃xff0c;否则所有的日志都会来进行匹配。grok本来就性能不好&＃xff0c;这么一搞&＃xff0c;机器就死掉了。grok {match &＃61;> { "message" &＃61;> "%{COMBINEDAPACHELOG}"}}
}output{elasticsearch {hosts &＃61;> ["10.0.0.204:9200"]index &＃61;> "apache-access-log-%{&＃43;YYYY.MM.dd}"}
}

效果

6. 项目实战

6.1 需求分析

访问日志&＃xff1a;apache访问日志&＃xff0c;nginx访问日志&＃xff0c;tomcat (file > filter)
错误日志&＃xff1a; java日志。
系统日志&＃xff1a;/var/log/* syslog rsyslog
运行日志&＃xff1a;程序写的&＃xff08;json格式&＃xff09;
网络日志&＃xff1a;防火墙、交换机、路由器。。。

标准化&＃xff1a;日志是什么格式的 json &＃xff1f;怎么命名&＃xff1f;存放在哪里&＃xff1f;/data/logs/? 日志怎么切割&＃xff1f;按天&＃xff1f;按小时&＃xff1f;用什么工具切割&＃xff1f;脚本&＃43;crontab ?
工具化&＃xff1a;如何使用logstash进行收集方案&＃xff1f;

6.2 实战

6.2.1 说明

收集日志&＃xff1a;nginx访问日志、apache访问日志、es日志、系统message日志

角色说明&＃xff1a;
10.0.0.203 web01-node1.damaiche.org-203 logstash 收集数据
10.0.0.204 elk01-node2.damaiche.org-204 logstash(indexer) &＃43; kibana &＃43; es &＃43; redis

6.2.2 客户端收集日志&＃xff0c;存放在redis上。

[root&＃64;web01-node1 conf.d]# cat shipper.conf
input{file {path &＃61;> "/var/log/nginx/access.log"start_position &＃61;> "beginning"type &＃61;> "nginx-access01-log"}file {path &＃61;> "/var/log/httpd/access_log"start_position &＃61;> "beginning"type &＃61;> "apache-access01-log"}file {path &＃61;> "/var/log/elasticsearch/myes.log"start_position &＃61;> "beginning"type &＃61;> "myes01-log"codec &＃61;> multiline {pattern &＃61;> "^\["negate &＃61;> "true"what &＃61;> "previous"}}file {path &＃61;> "/var/log/messages"start_position &＃61;> "beginning"type &＃61;> "messages01"}}output{if [type] &＃61;&＃61; "nginx-access01-log" {redis {host &＃61;> "10.0.0.204"port &＃61;> "6379"db &＃61;> "3"data_type &＃61;> "list"key &＃61;> "nginx-access01-log"}}if [type] &＃61;&＃61; "apache-access01-log" {redis {host &＃61;> "10.0.0.204"port &＃61;> "6379"db &＃61;> "3"data_type &＃61;> "list"key &＃61;> "apache-access01-log"}}if [type] &＃61;&＃61; "myes01-log" {redis {host &＃61;> "10.0.0.204"port &＃61;> "6379"db &＃61;> "3"data_type &＃61;> "list"key &＃61;> "myes01-log"}}if [type] &＃61;&＃61; "messages01" {redis {host &＃61;> "10.0.0.204"port &＃61;> "6379"db &＃61;> "3"data_type &＃61;> "list"key &＃61;> "messages01"}}
}[root&＃64;web01-node1 conf.d]# /etc/init.d/logstash start

注&＃xff1a;

• 1 启动过后&＃xff0c;需关注下logstash是否有异常输出。最好仔细核对下配置文件然后启动&＃xff0c;如有报错&＃xff0c;结合报错日志进行分析。ps:一般都是配置文件写错了&＃xff0c;引起的报错。
• 2 记得造日志啊。
• 3 logstash是用rpm安装的&＃xff0c;默认的启动用户是logstash,收集某些日志时&＃xff0c;可能权限不够无法正常收集。
  有2种解决方法:
  • 1 将logstash用户加入到root组.
  • 2.将logstsh用root用户启动&＃xff08;修改/etc/init.d/logstash 脚本&＃xff0c;将LS_USER改成root

登录redis查看各个key是否正常生成&＃xff0c;以及key的长度。

info
....
# Keyspace
db0:keys&＃61;1,expires&＃61;0,avg_ttl&＃61;0
db3:keys&＃61;4,expires&＃61;0,avg_ttl&＃61;0
db6:keys&＃61;5,expires&＃61;0,avg_ttl&＃61;010.0.0.204:6379> select 3
OK10.0.0.204:6379[3]> keys *
1) "messages01"
2) "myes01-log"
3) "apache-access01-log"
4) "nginx-access01-log"10.0.0.204:6379[3]> llen messages01
(integer) 2059610.0.0.204:6379[3]> llen myes01-log
(integer) 233610.0.0.204:6379[3]> llen apache-access01-log
(integer) 9265710.0.0.204:6379[3]> llen nginx-access01-log
(integer) 100820

6.2.3 es上启动一个logstash从redis里读取数据。

[root&＃64;elk01-node2 conf.d]# cat indexer.conf
input {redis {host &＃61;> "10.0.0.204"port &＃61;> "6379"db &＃61;> "3"data_type &＃61;> "list"key &＃61;> "nginx-access01-log"}redis {host &＃61;> "10.0.0.204"port &＃61;> "6379"db &＃61;> "3"data_type &＃61;> "list"key &＃61;> "apache-access01-log"}redis {host &＃61;> "10.0.0.204"port &＃61;> "6379"db &＃61;> "3"data_type &＃61;> "list"key &＃61;> "myes01-log"}redis {host &＃61;> "10.0.0.204"port &＃61;> "6379"db &＃61;> "3"data_type &＃61;> "list"key &＃61;> "messages01"}
}filter {if [type] &＃61;&＃61; "apache-access01-log" {grok {match &＃61;> { "message" &＃61;> "%{COMBINEDAPACHELOG}"}}}
}output {if [type] &＃61;&＃61; "nginx-access01-log" {elasticsearch {hosts &＃61;> ["10.0.0.204:9200"]index &＃61;> "nginx-access01-log-%{&＃43;YYYY.MM.dd}"}}if [type] &＃61;&＃61; "apache-access01-log" {elasticsearch {hosts &＃61;> ["10.0.0.204:9200"]index &＃61;> "apache-access01-log-%{&＃43;YYYY.MM.dd}"}}if [type] &＃61;&＃61; "myes01-log" {elasticsearch {hosts &＃61;> ["10.0.0.204:9200"]index &＃61;> "myes01-log-%{&＃43;YYYY.MM.dd}"} }if [type] &＃61;&＃61; "messages01" {elasticsearch {hosts &＃61;> ["10.0.0.204:9200"]index &＃61;> "messages01-%{&＃43;YYYY.MM}"} }
}

如果使用redis list作为elkstack的消息队列&＃xff0c;需要对所有的list key的长度进行监控。
根据实际情况&＃xff0c;例如超过&＃39;10w&＃39;就报警。