ELK(Elasticsearch+Filebeat+Kibana)轻量级采集分析Nginx日志

ELK是什么？

轻量级日志统计分析组件，包含elasticsearch、filebeat、kibana

ELK环境准备

Elasticsearch 下载地址
https://www.elastic.co/downloads/past-releases/elasticsearch-6-4-2
Elasticsearch 参考文档
https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
Filebeat 下载地址
https://www.elastic.co/downloads/past-releases/filebeat-6-4-2
Filebeat 参考文档
https://www.elastic.co/guide/en/beats/filebeat/6.4/index.html
Kibana 下载地址
https://www.elastic.co/downloads/past-releases/kibana-6-4-2
Kibana 参考文档
https://www.elastic.co/guide/en/kibana/6.4/index.html

ELK安装

下载elasticsearch-6.4.2.tar.gz
解压tar –zvxf elasticsearch-6.4.2.tar.gz
下载filebeat-6.4.2-linux-x86_64.tar.gz
解压 tar –zvxf filebeat-6.4.2-linux-x86_64.tar.gz
下载kibana-6.4.2-linux-x86_64.tar.gz
解压 tar –zvxf kibana-6.4.2-linux-x86_64.tar.gz

什么是Filebeat

如官网图片所示，Filebeat是一个轻量级的日志收集、过滤的中间件。可以向elasticsearch、logstash等推送数据。

Filebeat配置

• Input配置

filebeat.inputs:
# 文档类型
- type: log
# 是否开启 true or false
  enabled: true
# 日志路径
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

• 设置Kibana

setup.kibana:
  host: "kibanahost:5601"

• 设置output

output.elasticsearch:
  # es服务器配置
  hosts: ["Elasticsearchhost:9200"]
  # 指定索引文件名称
index: "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"

• Filebeat Nginx Module

Filebeat默认已经支持很多Module，通过官网可知如下

参考文档：https://www.elastic.co/guide/en/beats/filebeat/6.4/filebeat-module-nginx.html

Nginx moduleedit The nginx module parses access and error logs created
by the Nginx HTTP server.

When you run the module, it performs a few tasks under the hood:

Sets the default paths to the log files (but don’t worry, you can
override the defaults) Makes sure each multiline log event gets sent
as a single event Uses ingest node to parse and process the log lines,
shaping the data into a structure suitable for visualizing in Kibana
Deploys dashboards for visualizing the log data Compatibilityedit This
module requires the ingest-user-agent and ingest-geoip Elasticsearch
plugins.

The Nginx module was tested with logs from version 1.10.

On Windows, the module was tested with Nginx installed from the
Chocolatey repository.

官网已经明确说明，我们需要安装ingest-user-agent和ingest-geoip elasticsearch插件
这里说明一下，是要在elasticsearch下安装

安装ingest-user-agent
参考文档
https://www.elastic.co/guide/en/elasticsearch/plugins/6.4/ingest-user-agent.html
下载离线包
https://artifacts.elastic.co/downloads/elasticsearch-plugins/ingest-user-agent/ingest-user-agent-6.4.2.zip.
这里我们选择离线安装
https://www.elastic.co/guide/en/elasticsearch/plugins/6.4/plugin-management-custom-url.html

下载完成后通过以下命令安装插件

sudo bin/elasticsearch-plugin install file:///path/to/plugin.zip

file:///path/to/plugin.zip 为离线插件所在路径

安装ingest-geoip

参考文档

https://www.elastic.co/guide/en/elasticsearch/plugins/6.4/ingest-geoip.html
下载离线包
https://artifacts.elastic.co/downloads/elasticsearch-plugins/ingest-geoip/ingest-geoip-6.4.2.zip.
这里我们选择离线安装
https://www.elastic.co/guide/en/elasticsearch/plugins/6.4/plugin-management-custom-url.html

下载完成后通过以下命令安装插件

sudo bin/elasticsearch-plugin install file:///path/to/plugin.zip

file:///path/to/plugin.zip 为离线插件所在路径

• 开启nginx module

./filebeat modules enable nginx

通过以下命令查看是否开启成功

./filebeat modules list

• 配置nginx module
  进入/home/elk/filebeat-6.4.2-linux-x86_64/modules.d 路径下

vi nginx.yml

这里配置自己的access.log和error.log日志路径即可，这里是作者环境下的nginx日志路径。

启动Elasticsearch

运行

./bin/elasticsearch -d 

【需要创建非root账号运行】

chown -R elasticesearch:elasticesearch elasticsearch-6.4.2

这个时候可能发现除去本机以外的机器都无法访问,即使放开了9200端口也不行。解决方案如下：
https://www.jianshu.com/p/211982465c3b

启动Kibana

配置Kibana

设置当前IP

server.host: "Kibana Host"

后台运行

./bin/kibana &

启动filebeat

nohup ./filebeat -e -c filebeat.yml > filebeat.log &

启动 dashboard

./filebeat setup --dashboards

访问Kibana

一般是localhost:5601

Nginx Dashboard

找到Nginx模块

这里说明一下，如果部署和配置过程中有试错的情况
注意
• 启动了之后 filebeat.yml中 input 不需要改变
• 之前有数据模板不对 需要清空，否则无法使用

Ok 到了这里Nginx模块我们已经搭建完成。