ELK收集Nginx日志并进行展示

前段时间服务器遭受到攻击&＃xff0c;发现的时间比较晚&＃xff0c;于是我转行做起运维。解决方案是代码里面加了日志跟踪&＃xff0c;使用ELK&＃xff08;Elasticsearch&＃43;Logstash&＃43;Kibana&＃xff09;收集Nginx的日志作为监控。ELK在前东家的时候使用过&＃xff0c;觉得还是挺好用的&＃xff0c;现在自己尝试一下搭建起来。Nginx在docker上已经部署起来了&＃xff0c;接下来是采用docker把ELK搭建起来。Elasticsearch在大数据分析的时候听的比较多&＃xff0c;Logstash是负责收集Nignx日志然后推送到Elasticsearch&＃xff0c;Kibana负责展示这些数据&＃xff0c;接下来一起关注怎么把这三者搭建起来。

Elasticsearch在此不详细讲述了&＃xff0c;网络上有很多详细的资料。我们一起关注下通过docker怎么搭建Elasticsearch&＃xff0c;dockerhub的官网&＃xff1a;https://hub.docker.com/_/elasticsearch。具体的操作步骤如下&＃xff1a;

1. 拉取镜像&＃xff1a;docker pull Elasticsearch
2. 修改虚拟内存区域大小&＃xff1a;sysctl -w vm.max_map_count&＃61;262144
3. 修改elasticsearch.yml文件如下&＃xff1a;

cluster.name: "docker-cluster"
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"


这里要特别注意一点是&＃xff1a;key: value冒号和value中间是有空格的&＃xff01;&＃xff01;&＃xff01;

4. 启动Elasticsearch服务&＃xff0c;并挂载目录。

docker cp elasticsearch:/usr/share/elasticsearch/data /data/elk7/elasticsearch/
docker cp elasticsearch:/usr/share/elasticsearch/logs /data/elk7/elasticsearch/
chmod 777 -R /data/elk7/elasticsearch/docker rm -f elasticsearch
docker run -d --name&＃61;elasticsearch \--restart&＃61;always \-p 9200:9200 -p 9300:9300 \-e "discovery.type&＃61;single-node" \-v /data/elk7/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \-v /data/elk7/elasticsearch/data:/usr/share/elasticsearch/data \-v /data/elk7/elasticsearch/logs:/usr/share/elasticsearch/logs \elasticsearch:7.5.1


访问9200端口&＃xff0c;出现以下截图说明成功安装elasticsearch。

5. 启动elasticsearch head 插件

docker run -d \--name&＃61;elasticsearch-head \--restart&＃61;always \-p 9100:9100 \docker.io/mobz/elasticsearch-head:5-alpine


访问9100端口&＃xff1a;

1. 启动logstash

docker run -d --name&＃61;logstash logstash:7.5.1


2. 挂载目录

docker cp logstash:/usr/share/logstash /data/elk7/
mkdir /data/elk7/logstash/config/conf.d
chmod 777 -R /data/elk7/logstash


3. 修改配置文件

vim /data/elk7/logstash/config/logstash.yml


内容如下&＃xff1a;

http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://elasticseaerch:9200" ]
path.config: /usr/share/logstash/config/conf.d/*.conf
path.logs: /usr/share/logstash/logs


4. 新建syslog.congf文件&＃xff0c;用于收集/var/log/messages

vim /data/elk7/logstash/config/conf.d/syslog.conf


内容如下&＃xff1a;

input {file {#标签type &＃61;> "systemlog-localhost"#采集点path &＃61;> "/var/log/messages"#开始收集点start_position &＃61;> "beginning"#扫描间隔时间&＃xff0c;默认是1s&＃xff0c;建议5sstat_interval &＃61;> "5"}
}output {elasticsearch {hosts &＃61;> ["elasticsearch的ip:9200"]index &＃61;> "logstash-system-localhost-%{&＃43;YYYY.MM.dd}"}
}


4. 设置日志文件的读取权限

chmod 644 /var/log/messages


完成后访问9200端口&＃xff1a;

1. 拉取镜像

docker pull kibana:7.5.1


2. 修改配置文件

mkdir -p /data/elk7/kibana/config/
vi /data/elk7/kibana/config/kibana.yml


3. 启动容器

docker run -d \--name&＃61;kibana \--restart&＃61;always \-p 5601:5601 \-v /data/elk7/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml \kibana:7.5.1


访问5601端口&＃xff1a;

1. 设置elasticsearch的用户密码&＃xff0c;具体配置文件如下&＃xff1a;

cluster.name: "docker-cluster"
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

添加完成后需要设置进入容器&＃xff0c;到bin目录下运行elasticsearch-setup-passwords interactive。设置一系列用户的密码&＃xff1a;

Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana_system]:
Reenter password for [kibana_system]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]


设置完成后重启elasticsearch容器。

2. 设置kibana的用户密码&＃xff0c;具体配置文件如下&＃xff1a;

elasticsearch.username: "elastic"
elasticsearch.password: "********"
i18n.locale: "zh-CN"


完成后重启容器即可。

3. 配置logstash的用户密码。在配置文件下配置即可。

xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: "123123"

参考

1. ELK分析Nginx日志和可视化展示
2. docker安装Elasticsearch&＃43;Kibana&＃43;密码配置