基于AmbariHDP3.1.0.0群下安装KYLIN2.6.4集群

     基于ambari-2.7.3.0  、 HDP-3.1.0.0 、 HDP-UTILS-1.1.0.22 、HDP-GPL-3.1.0.0集群下手动安装KYLIN集群&＃xff08;hbase为2.0.2&＃xff09;。

1.下载并解压

yum -y install axel
axel -n 10 https://archive.apache.org/dist/kylin/apache-kylin-2.6.4/apache-kylin-2.6.4-bin-hadoop3.tar.gz
tar -zxvf apache-kylin-2.6.4-bin-hadoop3.tar.gz -C .
ln -s apache-kylin-2.6.4-bin-hadoop3 kylin

2.配置环境变量

vi /etc/profile
############################################################
#java
export JAVA_HOME&＃61;/opt/jdk8
export PATH&＃61;$JAVA_HOME/bin:$PATH#kafka
export KAFKA_HOME&＃61;/opt/kafka
export PATH&＃61;$KAFKA_HOME/bin:$PATH#hive
export HIVE_HOME&＃61;/usr/hdp/3.1.0.0-78/hive/
export HIVE_CONF&＃61;/usr/hdp/3.1.0.0-78/hive/conf
export PATH&＃61;$HIVE_HOME/bin:$PATH#hcat
export HCAT_HOME&＃61;/usr/hdp/3.1.0.0-78/hive-hcatalog
export PATH&＃61;$HCAT_HOME/bin:$PATH#spark
export SPARK_HOME&＃61;/opt/spark
export PATH&＃61;$PATH:$SPARK_HOME/bin#kylin
export KYLIN_HOME&＃61;/opt/kylin
export PATH&＃61;$KYLIN_HOME/bin:$PATH
#############################################################

3.配置tomcat/conf/server.xml

vi /opt/kylin/tomcat/conf/server.xml
############################################################
# 注释掉下面https

############################################################

4.加载环境变量

vim /opt/kylin/bin/header.sh
#################################
source /etc/profile
#################################

5.配置conf/kylin.properties

vi /opt/kylin/conf/kylin.properties
####################################################################
kylin.metadata.url&＃61;kylin_metadata&＃64;hbase
kylin.server.mode&＃61;all
kylin.server.cluster-servers&＃61;server1:7070,server2:7070,server3:7070,server4:7070,server5:7070
kylin.rest.servers&＃61;server1:7070,server2:7070,server3:7070,server4:7070,server5:7070
kylin.job.jar&＃61;/opt/kylin/lib/kylin-job-2.6.1.jar
kylin.coprocessor.local.jar&＃61;/opt/kylin/lib/kylin-coprocessor-2.6.1.jar
kylin.web.timezone&＃61;GMT&＃43;8
####################################################################

  备注&＃xff1a;

• kylin.server.mode 只有一个节点可以成为all或者job&＃xff08;用于任务构建&＃xff09;
• kyiln集群认为kylin.metadata.url为同一个即为同一个kyin集群

6.配置bin/find-hive-dependency.sh

vi /opt/kylin/bin/find-hive-dependency.sh
#################################################################
注释这一行&＃xff1a;hive_env&＃61;&＃96;hive ${hive_conf_properties} -e set 2>&1 | grep &＃39;env:CLASSPATH&＃39;&＃96;
添加&＃xff1a;hive -e set >/tmp/hive_env.txt 2>&1hive_env&＃61;&＃96;grep &＃39;env:CLASSPATH&＃39; /tmp/hive_env.txt&＃96;hive_env&＃61;&＃96;echo ${hive_env#*env:CLASSPATH}&＃96;hive_env&＃61;"env:CLASSPATH"${hive_env}
#################################################################

7.修改文件conf/kylin_job_conf.xml   conf/kylin_hive_conf.xml

分别在两个配置文件中注释以下两个属性
####################################

####################################

8.修改mapreducer.tar.gz包

# 删除hdfs上的老版本kafka-clients
sudo -u hdfs hadoop fs -copyToLocal /hdp/apps/3.1.0.0-78/mapreduce/mapreduce.tar.gz .
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
sudo -u hdfs hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.1.0.0-78/mapreduce/ # 删除本地老版本kafka-clients
cd /usr/hdp
find . -name &＃39;*kafka-clients*&＃39;
rm -rf kafka-clients-0.8.2.1.jar

9.安装对应版本的spark(如有则无需安装)

执行脚本自动下载安装&＃xff1a;
/opt/kylin/bin/download-spark.sh

10.确保以下服务启动

hadoop服务启动
jobhistory服务启动
hive元数据库启动
zookeeper服务启动
hbase服务启动

11.权限修改并启动

        因ambari安装的hadoop是hdfs用户&＃xff0c;kylin无权限写入&＃xff0c;所以要以hdfs用户启动

chown -R hdfs:hdfs /opt/kylinsudo -u hdfs /opt/kylin/bin/kylin.sh start

常见报错&＃xff1a;

   1.Something wrong with Hive CLI or Beeline, please execute Hive CLI or Beeline CLI in terminal to find the root cause.

    解决方案&＃xff1a;

          详见上面第六步:配置bin/find-hive-dependency.sh。 

  2.Caused by: java.lang.IllegalStateException: Cannot start job scheduler due to lack of job lock

    解决方案&＃xff1a;

          超过2个及以上kylin服务配置了kylin.server.mode&＃61;all&＃xff0c;只能有一个为all或者job。

   3.mkdir: Permission denied: user&＃61;root, access&＃61;WRITE, inode&＃61;"/":hdfs:hdfs:drwxr-xr-x
      Failed to create /kylin. Please make sure the user has right to access /kylin

     解决方案&＃xff1a;

           修改权限并以hdfs用户启动即可。

    4.kylin build构建时&＃xff0c;提示不能找到 kafka依赖

     解决方案&＃xff1a;

            详见上面第四步: 加载环境变量

     5. kylin 2.6.4 存在BUG 与 hdp 中的包冲突&＃xff0c; 无法从kafka构建&＃xff0c; 提示 partition.assignment.strategy 没有设置默认值。

         Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;

        Caused by: org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.engine.mr .exception.MapReduceException: no counters for job job_1570617080701_0005Job Diagnostics:Task failed task_1570617080701_0005_m_000000 

           详见上面第八步: 修改mapreducer.tar.gz包

      6. kylin2.6.4 在流式构建时提示某个hive属性不能使用

          Error: Could not open client transport for any of the Server URI&＃39;s in ZooKeeper: Failed to open new session: java.lang.IllegalArgumentException: Cannot modify dfs.replication at runtime. It is not in list of params that are allowed to be modified at runtime (state&＃61;08S01,code&＃61;0)
Cannot run commands specified using -e. No current connection

         详见上面第七步&＃xff1a;修改hive相关配置文件

外链&＃xff1a;

•   https://segmentfault.com/a/1190000011506398
•   https://blog.csdn.net/CREATE_17/article/details/89055969