sqoop2java_sqoop2安装总结

sqoop2安装

1. 下载解压缩

此次安装版本为1.99.6

# Decompress Sqoop distribution tarball

tar -xvf sqoop--bin-hadoop.tar.gz

ln -s sqoop--bin-hadoop.tar.gz sqoop

export SQOOP_HOME&＃61;&＃96;pwd&＃96;/sqoop

# Change working directory

cd $SQOOP_HOME

2. 配置服务

配置文件在$SQOOP_HOME/server/conf下&＃xff0c;大多数默认配置即可。需要注意的配置是catalina.properties文件下common_loader需要正确配置hadoop库文件&＃xff0c;包括hdfs、mapreduce、hive等所有jar包. 我的配置为:

common.loader&＃61;${catalina.base}/lib,\

${catalina.base}/lib/*.jar,\

${catalina.home}/lib,\

${catalina.home}/lib/*.jar,\

${catalina.home}/../lib/*.jar,\

/opt/cloudera/parcels/CDH/lib/hadoop/*.jar,\

/opt/cloudera/parcels/CDH/lib/hadoop/lib/*.jar,\

/opt/cloudera/parcels/CDH/lib/hadoop-hdfs/*.jar,\

/opt/cloudera/parcels/CDH/lib/hadoop-hdfs/lib/*.jar,\ /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/*.jar,\

/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*.jar,\

/opt/cloudera/parcels/CDH/lib/hadoop-yarn/*.jar,\

/opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib/*.jar,\

/opt/cloudera/parcels/CDH/lib/hive/lib/*.jar

sqoop.properties文件的org.apache.sqoop.submission.engine.mapreduce.configuration.directory&＃61;/etc/hadoop/conf/配置hadoop

配置文件路径&＃xff0c;默认为/etc/hadoop/conf

3. 测试配置是否正确

运行

sqoop2-tool verify

若正确,则输出为:

Verification was successful.

Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly

4. 下载derby 包

自带derby版本过低运行时会由于不兼容而出现以下错误&＃xff1a;

org.apache.sqoop.common.SqoopException: JDBCREPO_0007:Unable to lease link

at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:63)

at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:85)

at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:61)

at org.apache.sqoop.repository.JdbcRepository.createOrUpgradeRepository(JdbcRepository.java:127)

at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:123)

at org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:39)

at org.apache.sqoop.tools.ConfiguredTool.runTool(ConfiguredTool.java:35)

at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:75)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.sqoop.tomcat.TomcatToolRunner.main(TomcatToolRunner.java:77)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.catalina.startup.Tool.main(Tool.java:225)

Caused by: java.sql.SQLException: No suitable driver found for

解决办法为下载derby最新版本&＃xff0c;先删除/sqoop-1.99.6-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib下的derby旧包&＃xff0c;然后把新下载的derby目录下的lib下的jar包拷贝到/sqoop-1.99.6-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib

5. 验证

运行bin/sqoop2-shell,进入sqoop shell模式, 运行show version --all&＃xff0c;若能正确输出server版本&＃xff0c;则安装成功:

sqoop:000> show version --all

client version:

Sqoop 1.99.6 source revision 07244c3915975f26f03d9e1edf09ab7d06619bb8

Compiled by root on Wed Apr 29 10:40:43 CST 2015

server version:

Sqoop 1.99.6 source revision 07244c3915975f26f03d9e1edf09ab7d06619bb8

Compiled by root on Wed Apr 29 10:40:43 CST 2015

API versions:

[v1]