基于CentOS6.5或Ubuntu14.04下Suricata里搭配安装ELK(elasticsearch,logstash,kibana)（图文详解）...

作者： | 来源：互联网 | 2023-09-15 17:22

前期博客基于CentOS6.5下Suricata（一款高性能的网络IDS、IPS和网络安全监控引擎）的搭建（图文详解）&#x

　　前期博客

基于CentOS6.5下Suricata&＃xff08;一款高性能的网络IDS、IPS和网络安全监控引擎&＃xff09;的搭建&＃xff08;图文详解&＃xff09;&＃xff08;博主推荐&＃xff09;

基于Ubuntu14.04下Suricata&＃xff08;一款高性能的网络IDS、IPS和网络安全监控引擎&＃xff09;的搭建&＃xff08;图文详解&＃xff09;&＃xff08;博主推荐&＃xff09;

　　参考官网

https://redmine.openinfosecfoundation.org/projects/suricata/wiki/_Logstash_Kibana_and_Suricata_JSON_output

　　注意&＃xff0c;这官网&＃xff0c;默认是适合Ubuntu系统的安装手法。

　　

　　首先&＃xff0c;

　　Make sure your Suricata is compiled/installed with libjansson support enabled:

[root&＃64;suricata ~]# suricata --build-info
This is Suricata version 3.1 RELEASE
Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR&＃61;1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_NSS HAVE_LIBJANSSON TLS
SIMD support: SSE_4_2 SSE_4_1 SSE_3
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.4.7 20120313 (Red Hat 4.4.7-18), C version 199901
compiled with _FORTIFY_SOURCE&＃61;0
L1 cache line size (CLS)&＃61;64
thread local storage method: __thread
compiled with LibHTP v0.5.20, linked against LibHTP v0.5.20Suricata Configuration:AF_PACKET support: yesPF_RING support: noNFQueue support: noNFLOG support: noIPFW support: noNetmap support: noDAG enabled: noNapatech enabled: noUnix socket enabled: yesDetection enabled: yeslibnss support: yeslibnspr support: yeslibjansson support: yeshiredis support: noPrelude support: noPCRE jit: noLUA support: nolibluajit: nolibgeoip: noNon-bundled htp: noOld barnyard2 support: noCUDA enabled: noHyperscan support: noLibnet support: yesSuricatasc install: yesProfiling enabled: noProfiling locks enabled: noDevelopment settings:Coccinelle / spatch: noUnit tests enabled: noDebug output enabled: noDebug validation enabled: noGeneric build parameters:Installation prefix: /usr/localConfiguration directory: /usr/local/etc/suricata/Log directory: /usr/local/var/log/suricata/--prefix /usr/local--sysconfdir /usr/local/etc--localstatedir /usr/local/varHost: x86_64-pc-linux-gnuCompiler: gcc (exec name) / gcc (real)GCC Protect enabled: noGCC march native enabled: yesGCC Profile enabled: noPosition Independent Executable enabled: noCFLAGS -g -O2 -march&＃61;nativePCAP_CFLAGS SECCFLAGS
[root&＃64;suricata ~]#

　　然后&＃xff0c;需要配置

Configure suricata

[root&＃64;suricata suricata-3.1]# pwd
/root/suricata-3.1
[root&＃64;suricata suricata-3.1]# ls
aclocal.m4 compile config.h.in config.status configure.ac depcomp libhtp ltmain.sh Makefile.am qa scripts suricata.yaml
ChangeLog config.guess config.log config.sub contrib doc libtool m4 Makefile.in reference.config src suricata.yaml.in
classification.config config.h config.rpath configure COPYING install-sh LICENSE Makefile missing rules stamp-h1 threshold.config
[root&＃64;suricata suricata-3.1]# sudo vim suricata.yaml

# Extensible Event Format (nicknamed EVE) event log in JSON format- eve-log:enabled: yesfiletype: regular #regular|syslog|unix_dgram|unix_stream|redisfilename: eve.json#prefix: "&＃64;cee: " # prefix to prepend to each log entry# the following are valid when type: syslog above#identity: "suricata"#facility: local5#level: Info ## possible levels: Emergency, Alert, Critical,## Error, Warning, Notice, Info, Debug#redis:# server: 127.0.0.1# port: 6379# mode: list ## possible values: list (default), channel# key: suricata ## key or channel to use (default to suricata)# Redis pipelining set up. This will enable to only do a query every# &＃39;batch-size&＃39; events. This should lower the latency induced by network# connection at the cost of some memory. There is no flushing implemented# so this setting as to be reserved to high traffic suricata.# pipelining:# enabled: yes ## set enable to yes to enable query pipelining# batch-size: 10 ## number of entry to keep in buffertypes:- alert:# payload: yes # enable dumping payload in Base64# payload-buffer-size: 4kb # max size of payload buffer to output in eve-log# payload-printable: yes # enable dumping payload in printable (lossy) format# packet: yes # enable dumping of packet (without stream segments)http: yes # enable dumping of http fieldstls: yes # enable dumping of tls fieldsssh: yes # enable dumping of ssh fieldssmtp: yes # enable dumping of smtp fields# HTTP X-Forwarded-For support by adding an extra field or overwriting# the source or destination IP address (depending on flow direction)# with the one reported in the X-Forwarded-For HTTP header. This is# helpful when reviewing alerts for traffic that is being reverse# or forward proxied.xff:enabled: no# Two operation modes are available, "extra-data" and "overwrite".mode: extra-data# Two proxy deployments are supported, "reverse" and "forward". In# a "reverse" deployment the IP address used is the last one, in a# "forward" deployment the first IP address is used.deployment: reverse# Header name where the actual IP address will be reported, if more# than one IP address is present, the last IP address will be the# one taken into consideration.header: X-Forwarded-For- http:extended: yes # enable this for extended logging information# custom allows additional http fields to be included in eve-log# the example below adds three additional fields when uncommented#custom: [Accept-Encoding, Accept-Language, Authorization]- dns- tls:extended: yes # enable this for extended logging information- files:force-magic: no # force logging magic on all logged filesforce-md5: no # force logging of md5 checksums#- drop:# alerts: no # log alerts that caused drops- smtp:#extended: yes # enable this for extended logging information# this includes: bcc, message-id, subject, x_mailer, user-agent# custom fields logging from the list:# reply-to, bcc, message-id, subject, x-mailer, user-agent, received,# x-originating-ip, in-reply-to, references, importance, priority,# sensitivity, organization, content-md5, date#custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]# output md5 of fields: body, subject# for the body you need to set app-layer.protocols.smtp.mime.body-md5# to yes#md5: [body, subject]- ssh- stats:totals: yes # stats for all threads merged togetherthreads: no # per thread statsdeltas: no # include delta values# bi-directional flows- flow# uni-directional flows#- netflow

　　改为

# "United" event log in JSON format- eve-log:enabled: yestype: file #file|syslog|unix_dgram|unix_streamfilename: eve.json# the following are valid when type: syslog above#identity: "suricata" #facility: local5#level: Info ## possible levels: Emergency, Alert, Critical,## Error, Warning, Notice, Info, Debugtypes:- alert- http:extended: yes # enable this for extended logging information- dns- tls:extended: yes # enable this for extended logging information- files:force-magic: yes # force logging magic on all logged filesforce-md5: yes # force logging of md5 checksums#- drop- ssh- smtp

　　然后&＃xff0c;接下是安装

Install ELK (elasticsearch, logstash, kibana)

　　先卸载自带的openjdk

rpm -qa|grep java

rpm -e --nodeps tzdata-java-2013g-1.el6.noarch
rpm -e --nodeps java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64
rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64

　　注意&＃xff1a;这里一定要是jdk1.8&＃xff0c;不然会报如下错误

[root&＃64;suricata ~]# sudo /etc/init.d/elasticsearch start
Starting elasticsearch: Elasticsearch requires at least Java 8 but your Java version from /usr/local/jdk/jdk1.7.0_79/bin/java does not meet this requirement[FAILED]
[root&＃64;suricata ~]#

　　官网提示了&＃xff0c;建议安装orcal 的 jdk。

　　我这里安装jdk18。

[root&＃64;suricata suricata-3.1]# cd /usr/local/
[root&＃64;suricata local]# ls
bin etc games include lib lib64 libexec sbin share src var
[root&＃64;suricata local]# mkdir jdk
[root&＃64;suricata local]# cd jdk/
[root&＃64;suricata jdk]# pwd
/usr/local/jdk
[root&＃64;suricata jdk]# ll
total 0
[root&＃64;suricata jdk]# rz

[root&＃64;suricata jdk]# pwd
/usr/local/jdk
[root&＃64;suricata jdk]# ll
total 176992
-rw-r--r--. 1 root root 181238643 Sep 3 2016 jdk-8u60-linux-x64.tar.gz
[root&＃64;suricata jdk]# tar -zxvf jdk-8u60-linux-x64.tar.gz

　　然后&＃xff0c;再vim /etc/profile

#java
export JAVA_HOME&＃61;/usr/local/jdk/jdk1.8.0_60
export CLASSPATH&＃61;.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH&＃61;$PATH:$JAVA_HOME/bin

　　文件生效

source /etc/profile

　　然后&＃xff0c;接下来就是安装ELk和配置JDK。&＃xff08;注意以下只是个例子&＃xff09;

wget https://download.elasticsearch.org/kibana/kibana/kibana-3.0.0.tar.gz
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.6.1.deb
wget https://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.3-1_all.deb

tar -C /var/www/ -xzf kibana-3.0.0.tar.gz
dpkg -i elasticsearch-1.6.1.deb
dpkg -i logstash_1.5.3-1_all.deb

　　我这里啊&＃xff0c;Elasticsearch选取5.5.0

　　所以&＃xff0c;总的是

[root&＃64;suricata ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.deb

[root&＃64;suricata ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.0.deb

[root&＃64;suricata ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-5.5.0-linux-x86_64.tar.gz

　　最后下载&＃xff0c;得到

[root&＃64;suricata ~]# pwd
/root
[root&＃64;suricata ~]# ls
anaconda-ks.cfg Desktop Downloads install.log kibana-5.5.0-linux-x86_64.tar.gz Music Public suricata-3.1.tar.gz Videos
data Documents elasticsearch-5.5.0.deb install.log.syslog logstash-5.5.0.deb Pictures suricata-3.1 Templates
[root&＃64;suricata ~]#

　　然后&＃xff0c;进行对kibana的安装

　　得到

[root&＃64;suricata www]# pwd
/var/www
[root&＃64;suricata www]# ls
cgi-bin error html icons kibana-5.5.0-linux-x86_64
[root&＃64;suricata www]#

　　若大家没安装过这个命令&＃xff0c;则安装下即可。

[root&＃64;suricata ~]# yum -y install dpkg

　　然后&＃xff0c;进行对elasticsearch的安装

　　这里啊&＃xff0c;我建议大家还是去安装tar.gz包&＃xff0c;这种方式比较好。

　　所以&＃xff0c;后面&＃xff0c;我还是下载了&＃xff0c;rpm包&＃xff0c;以这种方式来进行安装。

[root&＃64;suricata ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.rpm

[root&＃64;suricata ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.0.rpm

　　　进行elasticsearch-5.5.0的安装

[root&＃64;suricata ~]# rpm -ivh elasticsearch-5.5.0.rpm
warning: elasticsearch-5.5.0.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing... ########################################### [100%]
Creating elasticsearch group... OK
Creating elasticsearch user... OK
1:elasticsearch ########################################### [100%]
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using chkconfig
sudo chkconfig --add elasticsearch
### You can start elasticsearch service by executing
sudo service elasticsearch start

　安装完&＃xff0c;默认即有

　　然后&＃xff0c;进行logstash的安装

[root&＃64;suricata ~]# pwd
/root
[root&＃64;suricata ~]# ls
anaconda-ks.cfg Desktop Downloads elasticsearch-5.5.0.rpm install.log.syslog logstash-5.5.0.deb Music Public suricata-3.1.tar.gz Videos
data Documents elasticsearch-5.5.0.deb install.log kibana-5.5.0-linux-x86_64.tar.gz logstash-5.5.0.rpm Pictures suricata-3.1 Templates
[root&＃64;suricata ~]# rpm -ivh logstash-5.5.0.rpm
warning: logstash-5.5.0.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing... ########################################### [100%]1:logstash ########################################### [100%]
Using provided startup.options file: /etc/logstash/startup.options
/usr/share/logstash/vendor/jruby/bin/jruby: line 388: /usr/bin/java: No such file or directory
/usr/share/logstash/vendor/jruby/bin/jruby: line 388: exec: /usr/bin/java: cannot execute: No such file or directory
Unable to install system startup script for Logstash.
[root&＃64;suricata ~]#

　　安装完&＃xff0c;默认即有

　　然后&＃xff0c;下面是

Logstash configuration

[root&＃64;suricata conf.d]# pwd
/etc/logstash/conf.d
[root&＃64;suricata conf.d]# ll
total 0
[root&＃64;suricata conf.d]# touch /etc/logstash/conf.d/logstash.conf

　　插入如下内容

input {file { path &＃61;> ["/var/log/suricata/eve.json"]sincedb_path &＃61;> ["/var/lib/logstash/"]codec &＃61;> json type &＃61;> "SuricataIDPS" }}filter {if [type] &＃61;&＃61; "SuricataIDPS" {date {match &＃61;> [ "timestamp", "ISO8601" ]}ruby {code &＃61;> "if event[&＃39;event_type&＃39;] &＃61;&＃61; &＃39;fileinfo&＃39;; event[&＃39;fileinfo&＃39;][&＃39;type&＃39;]&＃61;event[&＃39;fileinfo&＃39;][&＃39;magic&＃39;].to_s.split(&＃39;,&＃39;)[0]; end;" }}if [src_ip] {geoip {source &＃61;> "src_ip" target &＃61;> "geoip" #database &＃61;> "/opt/logstash/vendor/geoip/GeoLiteCity.dat" add_field &＃61;> [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]add_field &＃61;> [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]}mutate {convert &＃61;> [ "[geoip][coordinates]", "float" ]}if ![geoip.ip] {if [dest_ip] {geoip {source &＃61;> "dest_ip" target &＃61;> "geoip" #database &＃61;> "/opt/logstash/vendor/geoip/GeoLiteCity.dat" add_field &＃61;> [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]add_field &＃61;> [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]}mutate {convert &＃61;> [ "[geoip][coordinates]", "float" ]}}}}
}output { elasticsearch {host &＃61;> localhost#protocol &＃61;> http}
}

　　然后&＃xff0c;再

Configure the start-up services

update-rc.d elasticsearch defaults 95 10 &＃xff08;这两个是ubuntu系统里的做法&＃xff0c;我这里是centos系统&＃xff0c;不需要&＃xff09;
update-rc.d logstash defaults

service httpd start 启动apache
service elasticsearch start
service logstash start

或者

启动服务sudo /etc/init.d/elasticsearch start
停止服务sudo /etc/init.d/elasticsearch stop

启动服务sudo /etc/init.d/logstash/start
停止服务sudo /etc/init.d/logstashstop

　　如果大家在启动过程中&＃xff0c;出现如下问题。

[root&＃64;suricata ~]# sudo /etc/init.d/elasticsearch start
which: no java in (/sbin:/usr/sbin:/bin:/usr/bin)
Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME
[root&＃64;suricata ~]#

　　则见

ElasticSearch里JAVA环境丢失的问题

　　-rwxr-x---. 1 root root 5113 Jul 1 07:20 elasticsearch &＃xff0c;经过

[root&＃64;suricata init.d]# chmod 755 elasticsearch
[root&＃64;suricata init.d]# pwd
/etc/init.d
[root&＃64;suricata init.d]#

　　改为

-rwxr-xr-x. 1 root root 5113 Jul 1 07:20 elasticsearch

　　大家&＃xff0c;自行按照自己的jdk位置。

export JAVA_HOME&＃61;/usr/local/jdk/jdk1.8.0_60

　　然后&＃xff0c;就可以成功了。

[root&＃64;suricata ~]# sudo /etc/init.d/elasticsearch start
Starting elasticsearch: [ OK ]
[root&＃64;suricata ~]#

　　然后&＃xff0c;大家也许在启动&＃xff0c;会遇到这样的问题&＃xff1a;

Logstash默认的配置文件位置为./config和/etc/logstash/&＃xff0c;后者已经存在&＃xff0c;但直接运行依然会报错&＃xff1a;

　　WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash.

　　见

Logstash使用grok解析IIS日志

　　简单起见&＃xff0c;直接在Logstash根目录创建软链接即可&＃xff1a;

cd /usr/share/logstash

[root&＃64;suricata logstash]# pwd
/usr/share/logstash
[root&＃64;suricata logstash]# ln -s /etc/logstash ./config
[root&＃64;suricata logstash]#

　　设置配置文件实时生效&＃xff0c;从而不用频繁地启停Logstash。

　　修改/etc/logstash/logstash.yml

config.reload.automatic: true

　　Logstash的运行

　　与Elasticsearch、Kibana不同&＃xff0c;Logstash默认并不作为系统服务安装&＃xff0c;我们也不建议作为服务启动。主要原因为&＃xff1a;大多数情况下&＃xff0c;Elasticsearch、Kibana在每台服务器上只有一个实例&＃xff0c;可以作为服务启动&＃xff1b;而一个Logstash实例一般只服务于一个数据流&＃xff0c;因此一台服务器上经常运行多个实例。编写一个配置文件helloworld.conf进行测试。注意&＃xff0c;不要把该文件放在/etc/logstash/conf.d下&＃xff0c;以避免不同实例间产生混淆&＃xff0c;因为该文件夹是Logstash默认读取的文件夹。简单起见&＃xff0c;可直接放在Logstash根目录下。

input { stdin {}
}
output { stdout {}
}

　　这是一个命令行输入、命令行输出的Logstash实例&＃xff0c;运行成功则说明已配置正确&＃xff1a;

bin/logstash -f helloworld.conf

　　虽然如此&＃xff0c;很多时候我们还是希望各个Logstash实例能随系统而启动并在后台运行。将其加入计划任务即可&＃xff1a;

nohup /usr/share/logstash/bin/logstash -f /usr/share/logstash/helloworld.conf > /dev/null &