ELK日志分析系统（二）

【安装logstash并做一些日志搜集输出到elasticsearch中】

关闭防火墙关闭核心防护

systemctl stop firewalld.service
setenforce 0


更改主机名

hostnamectl set-hostname apache


安装Apahce服务(httpd)

[root&＃64;apache ~]# yum -y install httpd
[root&＃64;apache ~]# systemctl start httpd
[root&＃64;apche ~]# cd /var/log/httpd/
[root&＃64;apache httpd]# ls ##开启服务的时候才产生的日志
access_log error_log


安装Java环境

[root&＃64;apache ~]# java -version ####如果没有装安装yum -y install java
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)


附加

jdk java编译器 1.5 1.8 1.11
源文件--------->[jdk编译器]---------->执行文件----------->jvm(java虚拟机执行)
xxx.java xxx.class 结果手工编译部署
xxx.tar.gz(源码包)
c/c&＃43;&＃43;源码----------->[gcc/gcc-c&＃43;&＃43;编译器]&＃xff08;借助make工具&＃xff09;---------->执行文件----------->make install


安装logstash

上传logstash-5.5.1-rpm到根目录下
[root&＃64;apache ~]# cd /opt
[root&＃64;apache ~]# rpm -ivh logstash-5.5.1.rpm ##安装logstash
[root&＃64;apache ~]# systemctl start logstash.service ##启动logstash
[root&＃64;apache ~]# systemctl enable logstash.service
[root&＃64;apache ~]# In -s /usr/share/logstash/bin/logstash /usr/local/bin/ ##建立logstash软连接


附加

[root&＃64;apache ~]# rpm -qc logstash #查找配置文件位置
/etc/logstash/logstash.yml


输入采用标准输入输出采用标准输出

[root&＃64;apache ~]# logstash -e &＃39;input{ stdin{} } output { stdout{} }&＃39;
....&＃xff08;需要等待一会&＃xff09;
14:37:43.845 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port&＃61;>9600}
www.baidu.com #自行输入
2020-10-29T06:37:59.924Z apache www.baidu.com


使用rubydebug显示详细输出,codec为一种编解码器

[root&＃64;apache ~]# logstash -e &＃39;input{ stdin{} } output { stdout{ codec&＃61;>rubydebug } }&＃39;
....&＃xff08;需要等待一会&＃xff09;
14:47:48.735 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port&＃61;>9600}
www.baidu.com
{"&＃64;timestamp" &＃61;> 2020-10-29T06:47:54.671Z,"&＃64;version" &＃61;> "1","host" &＃61;> "apache","message" &＃61;> "www.baidu.com"
}


使用logstash将信息写入elasticsearch中

[root&＃64;apache ~]# logstash -e &＃39;input{ stdin{} } output { elasticsearch{ hosts&＃61;>["20.0.0.21:9200"] } }&＃39;
....&＃xff08;需要等待一会&＃xff09;
14:54:14.277 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port&＃61;>9600}
www.baidu.com


logstash配置文件
Logstash配置文件主要由三部分组成: input、output、filter(根据需要)

[root&＃64;apache ~]# chmod o&＃43;r /var/log/messages
[root&＃64;apache ~]# ll /var/log/messages
-rw----r--. 1 root root 817602 10月 29 15:03 /var/log/messages[root&＃64;apache opt]# vim /etc/logstash/conf.d/system.conf
input {file{path &＃61;> "/var/log/messages"type &＃61;> "system"start_position &＃61;> "beginning"}}
output {elasticsearch {hosts &＃61;> ["20.0.0.21:9200"]index &＃61;> "system-%{&＃43;YYYY.MM.dd}"}}[root&＃64;apache conf.d]# systemctl restart logstash.service


在主机安装kibana

上传kibana-5.5.1-x86_64.rpm 到/usr/local/src目录
[root&＃64;localhost ~]# cd /usr/local/src/
[root&＃64;localhost src]# rpm -ivh kibana-5.5.1-x86_64.rpm [root&＃64;localhost src]# cd /etc/kibana/
[root&＃64;localhost kibana]# cp kibana.yml kibana.yml.bak[root&＃64;localhost kibana]# vim kibana.yml
2/ server.port:5601 ####kibana打开的端口
7/ server.host:"0.0.0.0" ####kibana侦听的地址
21/ elasticsearch.url: "http://20.0.0.21:9200" ###和elasticsearch建立联系
30/ kibana.index: ".kibana" ###在elasticsearch中添加.kibana索引[root&＃64;localhost kibana]# systemctl start kibana.service ###启动kibana服务
[root&＃64;localhost kibana]# systemctl enable kibana.service ###开机启动kibana服务


使用浏览器输入http://20.0.0.12:5601/
首次登录创建一个索引名字:system-* ##这是对接系统日志文件
lndex name or pattern ##下面输入system-*
然后点最下面的出面的create按钮创建
然后点最左上角的Discover按钮会发现system-*信息
然后点下面的host旁边的add 会发现右面的图只有 Time 和 host 选项了这个比较友好

对接Apache主机的Apache日志文件&＃xff08;访问的、错误的)

[root&＃64;apache opt]# cd /etc/logstash/conf.d/
[root&＃64;apache conf.d]# touch apache_log.conf
[root&＃64;apache conf.d]# vim apache_log.conf
input {file{path &＃61;> "/etc/httpd/logs/access_log" ##access日志存放位置type &＃61;> "access" ##标签&＃xff08;可自行定义&＃xff09;start_position &＃61;> "beginning" ##从beginning开始}file{path &＃61;> "/etc/httpd/logs/error_log"type &＃61;> "error"start_position &＃61;> "beginning"}}
output {if [type] &＃61;&＃61; "access" {elasticsearch {hosts &＃61;> ["20.0.0.21:9200"]index &＃61;> "apache_access-%{&＃43;YYYY.MM.dd}"}}if [type] &＃61;&＃61; "error" {elasticsearch {hosts &＃61;> ["20.0.0.21:9200"]index &＃61;> "apache_error-%{&＃43;YYYY.MM.dd}"}}}[root&＃64;apache conf.d]# /usr/share/logstash/bin/logstash -f apache_log.conf


报错&＃xff1a;

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
15:57:32.317 [LogStash::Runner] ERROR logstash.inputs.file - Unknown setting &＃39;start_posotion&＃39; for file
15:57:32.324 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason&＃61;>"Something is wrong with your configuration."}


原因&＃xff1a;
something is wrong with your configuration
您的配置有问题

打开浏览器输入http://20.0.0.21:9100/查看索引信息

创建apache_error- 和 apache_access- 的索引**

创建完成