Hadoop集群(CDH4)实践之(3)Hive搭建

目录结构 Hadoop集群(CDH4)实践之 (0) 前言 Hadoop集群(CDH4)实践之 (1) Hadoop(HDFS)搭建 Hadoop集群(CDH4)实践之 (2) HBaseZookeeper搭建 Hadoop集群(CDH4)实践之 (3) Hive搭建 Hadoop集群(CHD4)实践之 (4) Oozie搭建 Hadoop集群(CHD4)实践之 (5) Sqoop安

目录结构
Hadoop集群(CDH4)实践之 (0) 前言
Hadoop集群(CDH4)实践之 (1) Hadoop(HDFS)搭建
Hadoop集群(CDH4)实践之 (2) HBase&Zookeeper搭建
Hadoop集群(CDH4)实践之 (3) Hive搭建
Hadoop集群(CHD4)实践之 (4) Oozie搭建
Hadoop集群(CHD4)实践之 (5) Sqoop安装

本文内容
Hadoop集群(CDH4)实践之 (3) Hive搭建

参考资料
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/CDH4-Installation-Guide.html

环境准备
OS: CentOS 6.4 x86_64
Servers:
hadoop-master: 172.17.20.230 内存10G
- namenode
- hbase-master

hadoop-secondary: 172.17.20.234 内存10G
- secondarybackupnamenode,jobtracker
- hive-server,hive-metastore

hadoop-node-1: 172.17.20.231 内存10G
- datanode,tasktracker
- hbase-regionserver,zookeeper-server

hadoop-node-2: 172.17.20.232 内存10G
- datanode,tasktracker
- hbase-regionserver,zookeeper-server

hadoop-node-3: 172.17.20.233 内存10G
- datanode,tasktracker
- hbase-regionserver,zookeeper-server

对以上角色做一些简单的介绍：
namenode - 整个HDFS的命名空间管理服务
secondarynamenode - 可以看做是namenode的冗余服务
jobtracker - 并行计算的job管理服务
datanode - HDFS的节点服务
tasktracker - 并行计算的job执行服务
hbase-master - Hbase的管理服务
hbase-regionServer - 对Client端插入，删除，查询数据等提供服务
zookeeper-server - Zookeeper协作与配置管理服务
hive-server - Hive的管理服务
hive-metastore - Hive的元存储，用于对元数据进行类型检查与语法分析

本文定义的规范，避免在配置多台服务器上产生理解上的混乱：
以下操作都只需要在 Hive 所在主机，即 hadoop-secondary 上执行。

1. 安装前的准备
Hadoop集群(CDH4)实践之 (2) HBase&Zookeeper搭建

2. 安装Hive
$ sudo yum install hive hive-metastore hive-server
$ sudo yum install hive-jdbc hive-hbase

3. 安装MySQL JDBC Connector
$ sudo yum install mysql-connector-java
$ sudo ln -s /usr/share/java/mysql-connector-java.jar /usr/lib/hive/lib/mysql-connector-java.jar

4. 安装MySQL
$ sudo yum install mysql-server
$ sudo /etc/init.d/mysqld start

$ sudo /usr/bin/mysql_secure_installation

 
[...]
Enter current password for root (enter for none):
OK, successfully used password, moving on...
[...]
Set root password? [Y/n] y
New password: hiveserver
Re-enter new password: hiverserver
Remove anonymous users? [Y/n] Y
[...]
Disallow root login remotely? [Y/n] N
[...]
Remove test database and access to it [Y/n] Y
[...]
Reload privilege tables now? [Y/n] Y
All done!

5. 创建数据库并授权
$ mysql -u root -phiveserver

 
mysql> CREATE DATABASE metastore;
mysql> USE metastore;
mysql> SOURCE /usr/lib/hive/scripts/metastore/upgrade/mysql/hive-schema-0.10.0.mysql.sql;
mysql> CREATE USER 'hive'@'%' IDENTIFIED BY 'hiveserver';
mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON metastore.* TO 'hive'@'%';
mysql> REVOKE ALTER,CREATE ON metastore.* FROM 'hive'@'%';
mysql> CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hiveserver';
mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON metastore.* TO 'hive'@'localhost';
mysql> REVOKE ALTER,CREATE ON metastore.* FROM 'hive'@'localhost';
mysql> CREATE USER 'hive'@'127.0.0.1' IDENTIFIED BY 'hiveserver';
mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON metastore.* TO 'hive'@'127.0.0.1';
mysql> REVOKE ALTER,CREATE ON metastore.* FROM 'hive'@'127.0.0.1';

6. 配置hive-site.xml
$ sudo vim /etc/hive/conf/hive-site.xml

 


  javax.jdo.option.ConnectionURL
  jdbc:mysql://hadoop-secondary/metastore
  the URL of the MySQL database
  javax.jdo.option.ConnectionDriverName
  com.mysql.jdbc.Driver
  javax.jdo.option.ConnectionUserName
  hive
  javax.jdo.option.ConnectionPassword
  hiveserver
  datanucleus.autoCreateSchema
  false
  datanucleus.fixedDatastore
  true
  datanucleus.autoStartMechanism 
  SchemaTable
  hive.metastore.uris
  thrift://hadoop-secondary:9083
  IP address (or fully-qualified domain name) and port of the metastore host
  hive.aux.jars.path
  file:////usr/lib/hive/lib/hbase.jar,file:///usr/lib/hive/lib/zookeeper.jar,file:///usr/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.5.0.jar,file:///usr/lib/hive/lib/guava-11.0.2.jar
  hbase.zookeeper.quorum
  hadoop-node-1,hadoop-node-2,hadoop-node-3

7. 启动Hive
$ /etc/init.d/hive-metastore start
$ /etc/init.d/hive-server start

8. 创建Hive所需的HDFS目录
$ sudo -u hdfs hadoop fs -mkdir /user/hive
$ sudo -u hdfs hadoop fs -mkdir /user/hive/warehouse
$ sudo -u hdfs hadoop fs -ls -R /user
$ sudo -u hdfs hadoop fs -chown -R hive /user/hive
$ sudo -u hdfs hadoop fs -chmod -R 1777 /user/hive/warehouse

$ sudo -u hdfs hadoop fs -chmod -R 777 /tmp/hadoop-mapred
$ sudo -u hdfs hadoop fs -chmod -R 777 /tmp/hive-hive
$ sudo chown -R hive:hive /var/lib/hive/.hivehistory

9. 至此，Hive的搭建就已经完成。