python存数据到impala_如何使用PythonImpyla客户端连接Hive和Impala

温馨提示&＃xff1a;要看高清无码套图&＃xff0c;请使用手机打开并单击图片放大查看。

1.文档编写目的

继上一章讲述如何在CDH集群安装Anaconda&搭建Python私有源后&＃xff0c;本章节主要讲述如何使用Pyton Impyla客户端连接CDH集群的HiveServer2和Impala Daemon&＃xff0c;并进行SQL操作。

内容概述

1.依赖包安装

2.代码编写

3.代码测试

测试环境

1.CM和CDH版本为5.11.2

2.RedHat7.2

前置条件

1.CDH集群环境正常运行

2.Anaconda已安装并配置环境变量

3.pip工具能够正常安装Python包

4.Python版本2.6&＃43; or 3.3&＃43;

5.非安全集群环境

2.Impyla依赖包安装

Impyla所依赖的Python包

six

bit_array

thrift (on Python 2.x) orthriftpy (on Python 3.x)

thrift_sasl

sasl

1.首先安装Impyla依赖的Python包

[root&＃64;ip-172-31-22-86~]# pip install bit_array[root&＃64;ip-172-31-22-86~]# pip install thrift&＃61;&＃61;0.9.3[root&＃64;ip-172-31-22-86~]# pip install six

[root&＃64;ip-172-31-22-86~]# pip install thrift_sasl

[root&＃64;ip-172-31-22-86~]# pip install sasl

注意&＃xff1a;thrift的版本必须使用0.9.3&＃xff0c;默认安装的为0.10.0版本&＃xff0c;需要卸载后重新安装0.9.3版本&＃xff0c;卸载命令pip uninstall thrift

2.安装Impyla包

impyla版本&＃xff0c;默认安装的是0.14.0&＃xff0c;需要将卸载后安装0.13.8版本

[root&＃64;ip-172-31-22-86ec2-user]# pip install impyla&＃61;&＃61;0.13.8CollectingimpylaDownloadingimpyla-0.14.0.tar.gz (151kB)100%|████████████████████████████████|153kB1.0MB/sRequirementalready satisfied: sixin/opt/cloudera/parcels/Anaconda-4.2.0/lib/python2.7/site-packages (fromimpyla)

Requirementalreadysatisfied: bitarrayin/opt/cloudera/parcels/Anaconda-4.2.0/lib/python2.7/site-packages (fromimpyla)

Requirementalreadysatisfied: thriftin/opt/cloudera/parcels/Anaconda-4.2.0/lib/python2.7/site-packages (fromimpyla)

Buildingwheelsforcollected packages: impylaRunningsetup.py bdist_wheelforimpyla ...doneStoredindirectory: /root/.cache/pip/wheels/96/fa/d8/40e676f3cead7ec45f20ac43eb373edc471348ac5cb485d6f5Successfullybuilt impylaInstallingcollected packages: impylaSuccessfullyinstalled impyla-0.14.0

3.编写Python代码

Python连接Hive&＃xff08;HiveTest.py&＃xff09;

fromimpala.dbapiimportconnect

conn&＃61;connect(host&＃61;&＃39;ip-172-31-21-45.ap-southeast-1.compute.internal&＃39;,port&＃61;10000,database&＃61;&＃39;default&＃39;,auth_mechan

ism&＃61;&＃39;PLAIN&＃39;)print(conn)cursor&＃61;conn.cursor()cursor.execute(&＃39;show databases&＃39;)printcursor.description# prints the result set&＃39;s schemaresults&＃61;cursor.fetchall()print(results)cursor.execute(&＃39;SELECT*FROM testlimit10&＃39;)printcursor.description# prints the result set&＃39;s schemaresults&＃61;cursor.fetchall()print(results)

Python连接Impala&＃xff08;ImpalaTest.py&＃xff09;

fromimpala.dbapiimportconnect

conn&＃61;connect(host&＃61;&＃39;ip-172-31-26-80.ap-southeast-1.compute.internal&＃39;,port&＃61;21050)print(conn)cursor&＃61;conn.cursor()cursor.execute(&＃39;show databases&＃39;)printcursor.description# prints the result set&＃39;s schemaresults&＃61;cursor.fetchall()print(results)cursor.execute(&＃39;SELECT*FROM testlimit10&＃39;)printcursor.description# prints the result set&＃39;s schemaresults&＃61;cursor.fetchall()print(results)

4.测试代码

在shell命令行执行Python代码测试

1.测试连接Hive

[root&＃64;ip-172-31-22-86ec2-user]# python HiveTest.py[(&＃39;database_name&＃39;, &＃39;STRING&＃39;, None, None, None, None, None)]

[(&＃39;default&＃39;,)]

[(&＃39;test.s1&＃39;, &＃39;STRING&＃39;,None, None, None, None, None),(&＃39;test.s2&＃39;, &＃39;STRING&＃39;, None, None, None, None, None)]

[(&＃39;name1&＃39;, &＃39;age1&＃39;),(&＃39;name2&＃39;, &＃39;age2&＃39;),(&＃39;name3&＃39;, &＃39;age3&＃39;),(&＃39;name4&＃39;, &＃39;age4&＃39;),(&＃39;name5&＃39;, &＃39;age5&＃39;),(&＃39;name6&＃39;, &＃39;age6&＃39;),(&＃39;name7&＃39;, &＃39;age7&＃39;),(&＃39;name8&＃39;, &＃39;age8&＃39;),(&＃39;name9&＃39;, &＃39;age9&＃39;),(&＃39;name10&＃39;, &＃39;age10&＃39;)][root&＃64;ip-172-31-22-86ec2-user]#

2.测试连接Impala

[root&＃64;ip-172-31-22-86ec2-user]# python ImpalaTest.py[(&＃39;name&＃39;, &＃39;STRING&＃39;, None, None, None, None, None),(&＃39;comment&＃39;, &＃39;STRING&＃39;, None, None, None, None, None)]

[(&＃39;_impala_builtins&＃39;, &＃39;Systemdatabase for Impala builtin functions&＃39;),(&＃39;default&＃39;, &＃39;Default Hive database&＃39;)]

[(&＃39;s1&＃39;, &＃39;STRING&＃39;, None, None, None,None, None),(&＃39;s2&＃39;, &＃39;STRING&＃39;, None, None, None,None, None)]

[(&＃39;name1&＃39;, &＃39;age1&＃39;),(&＃39;name2&＃39;, &＃39;age2&＃39;),(&＃39;name3&＃39;, &＃39;age3&＃39;),(&＃39;name4&＃39;, &＃39;age4&＃39;),(&＃39;name5&＃39;, &＃39;age5&＃39;),(&＃39;name6&＃39;, &＃39;age6&＃39;),(&＃39;name7&＃39;, &＃39;age7&＃39;),(&＃39;name8&＃39;, &＃39;age8&＃39;),(&＃39;name9&＃39;, &＃39;age9&＃39;),(&＃39;name10&＃39;, &＃39;age10&＃39;)][root&＃64;ip-172-31-22-86ec2-user]#

5.常见问题

1.错误一

building&＃39;sasl.saslwrapper&＃39;extensioncreatingbuild/temp.linux-x86_64-2.7creatingbuild/temp.linux-x86_64-2.7/saslgcc-pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Isasl -I/opt/cloudera/parcels/Anaconda/include/python2.7 -c sasl/saslwrapper.cpp -o build/temp.linux-x86_64-2.7/sasl/saslwrapper.ounableto execute&＃39;gcc&＃39;: No such file or directoryerror:command&＃39;gcc&＃39;failed with exit status1----------------------------------------

Command"/opt/cloudera/parcels/Anaconda/bin/python -u -c "import setuptools, tokenize;__file__&＃61;&＃39;/tmp/pip-build-kD6tvP/sasl/setup.py&＃39;;f&＃61;getattr(tokenize,&＃39;open&＃39;,open)(__file__);code&＃61;f.read().replace(&＃39;\r\n&＃39;,&＃39;\n&＃39;);f.close();exec(compile(code,__file__,&＃39;exec&＃39;))" install --record /tmp/pip-WJFNeG-record/install-record.txt --single-version-externally-managed --compile"failed with error code1in/tmp/pip-build-kD6tvP/sasl/

解决方法&＃xff1a;

[root&＃64;ip-172-31-22-86ec2-user]# yum -y install gcc[root&＃64;ip-172-31-22-86ec2-user]# yum install gcc-c&＃43;&＃43;

2.错误二

gcc-pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Isasl -I/opt/cloudera/parcels/Anaconda/include/python2.7 -c sasl/saslwrapper.cpp -o build/temp.linux-x86_64-2.7/sasl/saslwrapper.occ1plus:warning: command line option ‘-Wstrict-prototypes’isvalidforC/ObjC but notforC&＃43;&＃43; [enabled by default]Infile included from sasl/saslwrapper.cpp:254:0:sasl/saslwrapper.h:22:23:fatal error: sasl/sasl.h: No such file or directory#include ^

compilationterminated.error:command&＃39;gcc&＃39;failed with exit status1

解决方法&＃xff1a;

[root&＃64;ip-172-31-22-86ec2-user]# yum -y install python-devel.x86_64 cyrus-sasl-devel.x86_64

醉酒鞭名马&＃xff0c;少年多浮夸&＃xff01; 岭南浣溪沙&＃xff0c;呕吐酒肆下&＃xff01;挚友不肯放&＃xff0c;数据玩的花&＃xff01;

温馨提示&＃xff1a;要看高清无码套图&＃xff0c;请使用手机打开并单击图片放大查看。

您可能还想看

推荐关注Hadoop实操&＃xff0c;第一时间&＃xff0c;分享更多Hadoop干货&＃xff0c;欢迎转发和分享。

原创文章&＃xff0c;欢迎转载&＃xff0c;转载请注明&＃xff1a;转载自微信公众号Hadoop实操