ELK传统部署样例

1、ELK分布式日志平台的概念

ElasticSearch概念：

 ElasticSearch简称ES，主要是用于存储数据（日志内容）、分布式、可以检索日志内容、对日志内容进行过滤、可以从日志内容搜索相关的内容，支持自动发现节点、创建索引、支持副本集、丰富API接口（满足各种程序来使用ES）。

Logstash概念：

Logstash主要是部署在客户端节点上，主要是用于收集|采集，客户端服务器的相关的日志内容（内核日志、系统日志、安全日志、应用日志）的，除了可以采 集日志之外，还可以对日志进行简单过滤（过滤正则），最终会将收集的日志统 一存储至ES分布式服务器中。

Kibana概念：

Kibana是为ES和Logstash提供一套WEB工作界面的，用户通过WEB界面能够便 捷、高效的管理ES分布式集群、对日志快速匹配、操作，Kibana程序会连接ES 分布式引擎，可以操作分布式引擎中数据、日志。

FileBeat概念：

FileBeat跟Logstash是一样的功能，其特点是轻量级、高性能，部署在客户端节点上，主要是用于收集|采集，客户端服务器的相关的日志内容（内核日志、系统日志、安全日志、应用日志）的，但是其不能对日志内容进行过滤，如果需要 对日志进行过滤的话，需要将日志数据发送给Logstash，最终会将收集的日志 统一存储至ES分布式服务器中。

注意：Logstash和Elasticsearch是用Java语言编写，Filebeat是基于Go语言开发，而Kibana使用node.js框架，在配置ELK环境要保证系统有JAVA JDK开发库。  

2、部署操作：

传统部署步骤：
es和logstash上配置统一安装jdk1.8版本：

准备：安装jdk1.8
安装jdk
tar xf jdk1.8.0_131.tar.gz
mv jdk1.8.0_131 /usr/java/jdk1.8.0_131
vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_131
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH:$HOMR/bin

source /etc/profile

部署es

tar xf elasticsearch-5.3.0.tar.gz
mv elasticsearch-5.3.0 /usr/local/elasticsearch
vim /usr/local/elasticsearch/config/elasticsearch.yml
nethwork.host: 0.0.0.0
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
vim /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
vim /etc/sysctl.conf
vm.max_map_count=262144
sysctl -p
useradd elk
chown -R elk:elk /usr/local/elasticsearch/
su - elk
/usr/local/elasticsearch/bin/elasticsearch -d
ps -ef |grep elas
elk 1926 1 82 16:40 pts/0 00:00:16 /usr/java/jdk1.8.0_131/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissiOnsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/local/elasticsearch -cp /usr/local/elasticsearch/lib/elasticsearch-5.3.0.jar:/usr/local/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -d
elk 1966 1874 0 16:40 pts/0 00:00:00 grep --color=auto elas
netstat -tanpl |grep 9200
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::9200 :::* LISTEN 1926/java

部署kibana

安装kibana：
mv kibana-5.3.0-linux-x86_64 /usr/local/kibana
vim /usr/local/kibana/config/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://10.0.0.11:9200"
/usr/local/kibana/bin/kibana
log [08:55:46.247] [info][status][plugin:kibana@5.3.0] Status changed from uninitialized to green - Ready
log [08:55:46.340] [info][status][plugin:elasticsearch@5.3.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [08:55:46.387] [info][status][plugin:console@5.3.0] Status changed from uninitialized to green - Ready
log [08:55:46.624] [info][status][plugin:timelion@5.3.0] Status changed from uninitialized to green - Ready
log [08:55:46.628] [info][listening] Server running at http://0.0.0.0:5601
log [08:55:46.629] [info][status][ui settings] Status changed from uninitialized to yellow - Elasticsearch plugin is yellow
log [08:55:51.420] [info][status][plugin:elasticsearch@5.3.0] Status changed from yellow to yellow - No existing Kibana index found
log [08:55:52.478] [info][status][plugin:elasticsearch@5.3.0] Status changed from yellow to green - Kibana index ready
log [08:55:52.478] [info][status][ui settings] Status changed from yellow to green - Ready
访问页面：
默认访问页面出现即可：
logstash*引擎
timespace时间节点获取信息

部署logstash

部署logstash
tar xzf logstash-5.3.0.tar.gz
mv logstash-5.3.0 /usr/local/logstash/
mkdir -p /usr/local/logstash/config/etc/
cd /usr/local/logstash/config/etc/
vim jfedu.conf
input {
stdin { }
}
output {
stdout {
codec => rubydebug {}}
elasticsearch {
hosts => "10.0.0.11:9200" }
}
/usr/local/logstash/bin/logstash -f jfedu.conf
/usr/local/logstash/bin/logstash -f jfedu.conf
Sending Logstash's logs to /usr/local/logstash/logs which is now configured via log4j2.properties
[2022-04-05T17:04:18,848][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/local/logstash/data/queue"}
[2022-04-05T17:04:18,959][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"6a99e592-bb18-4519-87bf-3b1c59a376d6", :path=>"/usr/local/logstash/data/uuid"}
[2022-04-05T17:04:20,915][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://10.0.0.11:9200/]}}
[2022-04-05T17:04:20,930][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.0.0.11:9200/, :path=>"/"}
[2022-04-05T17:04:21,183][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#http://10.0.0.11:9200/>}
[2022-04-05T17:04:21,232][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2022-04-05T17:04:21,411][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword"}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2022-04-05T17:04:21,455][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
[2022-04-05T17:04:21,823][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#]}
[2022-04-05T17:04:21,868][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125}
[2022-04-05T17:04:21,941][INFO ][logstash.pipeline ] Pipeline main started
The stdin plugin is now waiting for input:
[2022-04-05T17:04:22,102][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
www.jf.com #########这里手动输入
{
"@timestamp" => 2022-04-05T09:07:26.034Z,
"@version" => "1",
"host" => "k8s-node-2",
"message" => "www.jf.com"
}
test #############手动输入
{
"@timestamp" => 2022-04-05T09:09:13.727Z,
"@version" => "1",
"host" => "k8s-node-2",
"message" => "test"
}
localhsot:9200timespace ######手动输入
{
"@timestamp" => 2022-04-05T09:09:57.272Z,
"@version" => "1",
"host" => "k8s-node-2",
"message" => "localhsot:9200timespace"
}
0

 测试效果看测试页面

成年人的崩溃，往往只是一瞬间，愿余生多乐趣！！！