SparkSql安装

一、启动zk、启动hdfs集群、启动hive服务&＃xff08;hive  --service  metastore&＃xff09;

二、修改spark中conf下的hive-site.xml配置文件&＃xff08;就和hive的client的配置文件相同&＃xff09;

    注&＃xff1a;这里配置文件只需要修改一台主机就可以了&＃xff0c;这里相当于client

三、启动spark集群

四、启动spark sql(./bin/spark-sql --master spark://node11:7077 --executor-memory 512m)

    注&＃xff1a;配置文件spark-env.sh中如果配置的是ip 那么命令就需要用ip&＃xff0c;如果配置文件中配置的是主机名&＃xff0c;那么命令就要用主机名&＃xff1b;

SparkSQL thrift server环境搭建

1、hive-site.xml 中添加配置

hive.server2.thrift.min.worker.threads

5

Minimum number of Thrift worker threads

hive.server2.thrift.max.worker.threads

500

Maximum number of Thrift worker threads

hive.server2.thrift.port

10000

Port number of HiveServer2 Thrift interface. Can be overridden by setting $HIVE_SERVER2_THRIFT_PORT

hive.server2.thrift.bind.host

node12

Bind host on which to run the HiveServer2 Thrift interface.Can be overridden by setting$HIVE_SERVER2_THRIFT_BIND_HOST

二、启动spark thrift server

./sbin/start-thriftserver.sh --master spark://192.168.57.4:7077 --executor-memory 512M

三、启动以后可以通过bin目录下beeline访问

./bin/beeline

!connect  jdbc:hive2://node12:10000

注&＃xff1a;这样可以进入sparksql控制台&＃xff0c;但是不能查询&＃xff0c;报错&＃xff0c;需要将hdfs集群中core-site.xml 和 hdfs-site.xml拷贝到spark中conf下&＃xff08;每个spark集群节点都要拷贝&＃xff09;