CentOS7安装Sqoop2连接MySQL(Hadoop2.6,MySQL6.5,Sqoop1.99.6)

为什么强调1.99.6因为不同Sqoop版本对应不同的Hadoop版本.

1.Hadoop环境,参考:

http://blog.csdn.net/a237428367/article/details/50462858

2.MySQL,参考:

http://blog.csdn.net/a237428367/article/details/50481719

3.tar -zxvf sqoop-1.99.6-bin-hadoop200.tar.gz 你懂的

   mv sqoop-1.99.6-bin-hadoop200 sqoop-1.99.6 重命名一下因为好长

4.vi /hadoop/sqoop-1.99.6/server/conf/sqoop.properties 修改hadoop配置文件所属路径

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/hadoop/hadoop-2.6.3/etc/hadoop/

5. vi /hadoop/sqoop-1.99.6/server/conf/catalina.properties 把/hadoop/hadoop-2.6.3/share/hadoop下所有*.jar包加入到类路径里.

这个是原来的配置:

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/lib/hadoop/*.jar,/usr/lib/hadoop/lib/*.jar,/usr/lib/hadoop-hdfs/*.jar,/usr/lib/hadoop-hdfs/lib/*.jar,/usr/lib/hadoop-mapreduce/*.jar,/usr/lib/hadoop-mapreduce/lib/*.jar,/usr/lib/hadoop-yarn/*.jar,/usr/lib/hadoop-yarn/lib/*.jar,/usr/lib/hive/lib/*.jar

这个是修改后的配置:

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/hadoop/hadoop-2.6.3/share/hadoop/common/*.jar,/hadoop/hadoop-2.6.3/share/hadoop/common/lib/*.jar,/hadoop/hadoop-2.6.3/share/hadoop/hdfs/*.jar,/hadoop/hadoop-2.6.3/share/hadoop/hdfs/lib/*.jar,/hadoop/hadoop-2.6.3/share/hadoop/mapreduce/*.jar,/hadoop/hadoop-2.6.3/share/hadoop/mapreduce/lib/*.jar,/hadoop/hadoop-2.6.3/share/hadoop/tools/*.jar,/hadoop/hadoop-2.6.3/share/hadoop/tools/lib/*.jar,/hadoop/hadoop-2.6.3/share/hadoop/yarn/*.jar,/hadoop/hadoop-2.6.3/share/hadoop/yarn/lib/*.jar,/hadoop/hadoop-2.6.3/share/hadoop/httpfs/tomcat/lib/*.jar

6. vi /etc/profile 环境变量

export JAVA_HOME=/usr/java/jdk1.8.0_65
export HADOOP_HOME=/hadoop/hadoop-2.6.3
export SQOOP_HOME=/hadoop/sqoop-1.99.6
export PATH=$PATH:${HADOOP_HOME}/bin:${SQOOP_HOME}/bin
export CATALINA_HOME=${SQOOP_HOME}/server
export LOGDIR=${SQOOP_HOME}/logs

7.MySQL JDBC:

http://dev.mysql.com/downloads/connector/j/

把mysql-connector-java-5.1.38-bin.jar 复制到 /hadoop/sqoop-1.99.6/server/lib 下

8.sqoop.sh server start 启动sqoop server,输出:

Sqoop home directory: /hadoop/sqoop-1.99.6
Setting SQOOP_HTTP_PORT:     12000
Setting SQOOP_ADMIN_PORT:     12001
Using   CATALINA_OPTS:
Adding to CATALINA_OPTS:    -Dsqoop.http.port=12000 -Dsqoop.admin.port=12001
Using CATALINA_BASE:   /hadoop/sqoop-1.99.6/server
Using CATALINA_HOME:   /hadoop/sqoop-1.99.6/server
Using CATALINA_TMPDIR: /hadoop/sqoop-1.99.6/server/temp
Using JRE_HOME:        /usr/java/jdk1.8.0_65
Using CLASSPATH:       /hadoop/sqoop-1.99.6/server/bin/bootstrap.jar

9. ./sqoop.sh client 进shell,输出:

sqoop:000>

10. set server --host master --port 12000 --webapp sqoop 为客户端配置服务器,输出:

Server is set successfully

11.创建MySQL连接:

sqoop:000> create link --cid 1
Creating link for connector with id 1
Please fill following values to create new link object
Name: MySQL207

Link configuration

JDBC Driver Class: com.mysql.jdbc.Driver
JDBC Connection String: jdbc:mysql://192.168.30.207:3306/mysql
Username: root
Password: ***********
JDBC Connection Properties:
There are currently 0 values in the map:
entry# protocol=tcp

ps.如果你需要删除这个link ,则 : delete link -l 1 (1这个id 从 show link 这个命令查出来)

Link成功则输出:

link was successfully updated with status OK

show link 看一下

+----+----------+--------------+------------------------+---------+
| Id |   Name   | Connector Id |     Connector Name     | Enabled |
+----+----------+--------------+------------------------+---------+
| 2  | MySQL207 | 1            | generic-jdbc-connector | true    |
+----+----------+--------------+------------------------+---------+

12. exit 退出client
     sqoop.sh server stop 停止sqoop server