基于CentOS的HBase，Hive，Kylin分布式集群搭建过程

一. HBase环境搭建

1. 编辑/opt/hbase-1.3.1/conf/hbase-env.sh&＃xff0c;如下所示&＃xff1a;

export JAVA_HOME&＃61;/usr/local/jdk1.8.0_141
export HBASE_HOME&＃61;/opt/hbase-1.3.1
export HBASE_LOG_DIR&＃61;/home/ssw/Data/hbase-logs
export HBASE_MANAGES_ZK&＃61;true

2. 编辑/opt/hbase-1.3.1/conf/hbase-site.xml&＃xff0c;如下所示&＃xff1a;



hbase.rootdir
hdfs://master:9000/hbase


hbase.cluster.distributed
true


hbase.master
master:60000


hbase.zookeeper.quorum
master,slave1,slave2


hbase.zookeeper.property.dataDir
/home/ssw/Data/zookeeper



3. 编辑/opt/hbase-1.3.1/conf/regionservers&＃xff0c;如下所示&＃xff1a;

Slave1
Slave2

4. 启动HBase集群

/opt/hbase-1.3.1/bin/start-hbase.sh

5. 查看集群是否启动成功

Master&＃xff1a;HQuorumPeer&＃xff1b;HMaster。Slave&＃xff1a;HQuorumPeer&＃xff1b;HRegionServer。

二. Hive环境搭建

1. 安装MySQL

wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
sudo yum install mysql-server
sudo chown -R root:root /var/lib/mysql
service mysqld restart
$ mysql -u root
mysql > use mysql;
mysql > update user set password&＃61;password(&＃39;root&＃39;) where user&＃61;&＃39;root&＃39;;
mysql > exit;
service mysqld restart

2. 为Hive建立相应的MySQL账户&＃xff0c;并赋予足够的权限&＃xff0c;如下所示&＃xff1a;

create user &＃39;hive&＃39; identified by &＃39;mysql&＃39;;
grant all privileges on *.* to &＃39;hive&＃39;&＃64;&＃39;%&＃39; with grant option;
flush privileges;
service mysqld restart


3. 建立Hive专用元数据库&＃xff0c;用创建的hive账号登录&＃xff0c;如下所示&＃xff1a;

mysql -uhive -p&＃39;mysql&＃39;
mysql> create database hive;

4. 加载MySQL驱动包

将mysql-connector-java-5.1.44-bin.jar放$HIVE_HOME$/lib目录下面。

5. 编辑/etc/profile&＃xff0c;如下所示&＃xff1a;

export HIVE_HOME&＃61;/opt/hive-2.3.0
export PATH&＃61;$PATH:$HIVE_HOME/bin

6. 编辑hive-config.sh&＃xff0c;如下所示&＃xff1a;

export JAVA_HOME&＃61;/usr/local/jdk1.8.0_141
export HIVE_HOME&＃61;/opt/hive-2.3.0
export HADOOP_HOME&＃61;/opt/hadoop-2.7.3

7. 编辑hive-site.xml&＃xff0c;如下所示&＃xff1a;


javax.jdo.option.ConnectionURL
jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist&＃61;true


javax.jdo.option.ConnectionDriverName
com.mysql.jdbc.Driver


javax.jdo.option.ConnectionUserName
hive


javax.jdo.option.ConnectionPassword
mysql


8. 编辑hive-env.sh&＃xff0c;如下所示&＃xff1a;

export HOME&＃61;&＃61;/opt/hadoop-2.7.3
export HIVE_CONF_DIR&＃61;/opt/hive-2.3.0/conf
export HIVE_AUX_JARS_PATH&＃61;/opt/hive-2.3.0/lib

9. 验证Hive配置是否成功&＃xff0c;如下所示&＃xff1a;

hive> create table test_table (id int, name string, no int);
mysql -uhive -p&＃39;mysql&＃39;; show databases; use hive; show tables; select * from TBLS;

说明&＃xff1a;Hive将元数据库存储在MySQL中&＃xff0c;元数据包括表的名字&＃xff0c;表的列和分区及其属性&＃xff0c;表的属性(是否为外部表等)&＃xff0c;表的数据所在目录。

三. Kylin环境搭建

四. 遇到的问题

1. create directory /tmp/hive/XXX. Name node is in safe mode. 

解析&＃xff1a;hdfs dfsadmin -safemode leave

2. FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 

Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.

解析&＃xff1a;用hive -hiveconf hive.root.logger&＃61;DEBUG,console排查问题。

3. org.apache.hadoop.hive.metastore.api.MetaException: Version information not found in metastore.

参考文献&＃xff1a;

[1] Kylin安装配置实验&＃xff1a;http://blog.csdn.net/wzy0623/article/details/51283352

[2] Access denied for user &＃39;hive&＃39;&＃64;&＃39;localhost&＃39; &＃xff1a;http://blog.csdn.net/adminyou/article/details/48328417

[3] Hive完全分布式集群安装过程&＃xff08;元数据库:MySQL&＃xff09;&＃xff1a;http://www.aboutyun.com/thread-6902-1-1.html

[4] Hive常见问题汇总&＃xff1a;http://blog.csdn.net/freedomboy319/article/details/44828337