作者:詹建红_335 | 来源:互联网 | 2023-08-23 11:49
标题有点噱头,不过网络环境好的情况下也差不多了^_^
1. 首先保证安装了jdk。
elasticsearch, logstash, kibana,filebeat都可以通过yum安装,这里前三者通过直接下载压缩包安装启动,filebeat通过yum安装。
2. 下载elasticsearch-5.1.1解压, 配置
elasticsearch-5.1.1/conf/elasticsearch.yml
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
启动(可以放到supervisor下监管):
elasticsearch-5.1.1/bin/elasticsearch
注意: es启动要求提高一些系统参数配置,否则会报错
a. 增大vm.max_map_count到至少262144
sudo vim /etc/sysctl.conf
添加 vm.max_map_count=262144
sudo sysctl -p
b. 增大文件句柄数至少 65536 ulimit -a查看
sudo vim /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
2. 下载logstash5.1.1解压, 配置
logstash-5.1.1/conf.d/pro-log.conf
input {
beats {
port => 5044
}
}
filter {
if [fields][logIndex] == "nginx" {
grok {
patterns_dir => "/home/elk/apps/logstash-5.1.1/patterns"
match => {
"message" => "%{NGINXACCESS}"
}
}
urldecode {
charset => "UTF-8"
field => "url"
}
if [upstreamtime] == "" or [upstreamtime] == "null" {
mutate {
update => { "upstreamtime" => "0" }
}
}
date {
match => ["logtime", "dd/MMM/yyyy:HH:mm:ss Z"]
target => "@timestamp"
}
mutate {
cOnvert=> {
"responsetime" => "float"
"upstreamtime" => "float"
"size" => "integer"
}
remove_field => ["port","logtime","message"]
}
}
}
output {
elasticsearch {
hosts => "{your-es-ip}:9200"
manage_template => false
index => "%{[fields][logIndex]}-%{+YYYY.MM.dd}"
document_type => "%{[fields][docType]}"
}
}
这里使用grok解析nginx日志
nginx日志格式:
log_format app_log_format '[$time_local] $server_addr $remote_addr $body_bytes_sent $request_time $upstream_response_time '
'$upstream_addr $upstream_status "$request_uri" "$http_x_forwarded_for" "$http_referer" "$http_user_agent" $status';
配置grok的自定义pattern(可以使用grok debugger工具进行验证 http://grokdebug.herokuapp.com/):
vim logstash-5.1.1/patterns/nginx
NGINXACCESS \[%{HTTPDATE:logtime}\] %{IPORHOST:host} %{IPORHOST:remoteaddr} (?:%{NUMBER:size}|-) %{NUMBER:responsetime} (?:%{NUMBER:upstreamtime}|-) %{URIHOST:upstreamhost} %{BASE10NUM:upstreamstatus} %{QS:url} %{QS:clientip} %{QS:referrer} %{QS:agent} %{INT:status}
启动(可以放到supervisor下监管):
logstash-5.1.1/bin/logstash -f logstash-5.1.1/conf.d/pro-log.conf
3. 安装filebeat,filebeat可以直接使用yum安装。
配置yum源:
vim /etc/yum.repos.d/elastic5.repo
[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
安装
sodu yum install filebeat
配置(默认开始output到es,需要注释掉)
sudo vim /etc/filebeat/filebeat.yml
filebeat.prospectors:
# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.
- input_type: log
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /opt/nginx/logs/app.access.log
fields:
logIndex: nginx
docType: nginx-access
project: app-nginx
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["{your-logstash-ip}:5044"]
启动
sudo systemctl start filebeat
4. 下载kibana-5.1.1解压,配置
kibana-5.1.1/config/kibana.yml
server.port: 5601
server.host: 0.0.0.0
elasticsearch.url: "http://{your-es-ip}:9200"
启动(可以放到supervisor下监管)
kibana-5.1.1-linux-x86_64/bin/kibana
5. (选择性安装)安装x-pack,x-pack包含了security(需要用户名密码访问kibana)、watcher(监控报警)等插件
elasticsearch.5.1.1/bin/elasticsearch-plugin install x-pack (很慢,最好可以通过VPN安装,或者通过下载x-pack zip包,通过离线安装)
离线安装:
elasticsearch.5.1.1/bin/elasticsearch-plugin install file:///home/elk/apps/x-pack-5.1.1.zip
kibana-5.1.1/bin/kibana-plugin install file:///home/elk/apps/x-pack-5.1.1.zip
装完后,重启es,kibana生效。
不让某个插件生效,如不让security生效,则在es与kibana的配置里加入
xpack.security.enabled: false
最后贴两张kibana统计查询nginx日志得出的api调用次数,及平均响应时间图表。