ES项目实战

前置

ES: Java
Spark/Flink Stack + Spring Boot + ES
Scala/Java + Java/Scala + Java
==> 用API的方式来掌握ES的用法(API、SpringBoot的使用)
ES: API RESTFul

ElasticSearch + Kibana
存储 展示/分析
ES Plugin: Head SQL Kibana (三个插件,Kibana也算插件)
最终数据要到SQL (易用性)

ES的安装

地址: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.2.tar.gz
?
?

• 解压: tar -zxvf elasticsearch-6.6.2.tar.gz -C ~/app/


• 配置到系统环境变量:


• 修改:
  cd elasticsearch-6.6.2
  删除bin目录中以bat结尾的命令(windows中的,无用)

$ES_HOME (重要的文件)

bin

elasticsearch 启动ES

elasticsearch 前台方法启动
elasticsearch -d 后台方法启动 (需要打开日志查看系统输出信息)
elasticsearch-plugin 管理ES插件

elasticsearch-sql-cli sql客户端

config

elasticsearch.yml es的配置信息

#cluster.name: my-application (集群名称)
#node.name: node-1 (节点名称)
#path.data: /path/to/data
Path to log files:
path.logs: /path/to/logs
#network.host: 0.0.0.0 (全网)
Set a custom port for HTTP:
http.port: 9200



jvm.optiom es的JVM相关的配置信息

## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms1g
-Xmx1g
## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
# explicitly set the stack size
-Xss1m
# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=data
# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=logs/hs_err_pid%p.log
# ----------------------------------- Memory -----------------------------------
elasticsearch-migrate x-pack-env
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:logs/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m


***注:ES对硬盘要求极高(一般SSD、内存大) ***

后台启动ES: elasticsearch -d

去Web UI查看: hadoop000:9200 (Web UI port)
hadoop000:9300 服务器端口(server port)
Lucene_version: "7.6.0" 不对应

技巧:用chrome中的json字符串美化插件 (JSON Formatter)

ES核心概念

Cluster
Node
Index Database
Type Table
Document Row
Field Column //这四种的REST API在工作中用的最多
shard(分片--分区集叫分片)
replica(副本)
对应关系:
Index -> Type -> Document -> Field
Database -> Table -> Row -> Column


代码开发

需要: IDEA+Maven+Java
pom.xml



org.elasticsearch.client
transport
6.6.2



junit
junit
4.11




PreBuiltTransportClient.java

/**
* Creates a new transport client with pre-installed plugins.
*
* @param settings the settings passed to this transport client
* @param plugins an optional array of additional plugins to run with this client
*/
@SafeVarargs
public PreBuiltTransportClient(Settings settings, Class... plugins) {
this(settings, Arrays.asList(plugins));
}
//Class...
// ...代表一个/多个可变参数, 可填可不填


TransportClient.java

/**
* Adds a transport address that will be used to connect to.
*


* The Node this transport address represents will be used if its possible to connect to it.
* If it is unavailable, it will be automatically connected to once it is up.
*


* In order to get the list of all the current connected nodes, please see {@link #connectedNodes()}.
*/
public TransportClient addTransportAddress(TransportAddress transportAddress) {
nodesService.addTransportAddresses(transportAddress);
return this;
}