在Windows环境中优化提交Spark任务的方法与实践

一、本地代码编译

mvn clean&＃160; install -DskipTests

二、构建spark提交脚本

样例如下：ip_spark.bat

@echo off
set param=%1
set PROJECT_DIR="D:\gitlab\CP_WiseCloudLocationService_WiseCloudGeoDataBI\geodata-dataprocess\geodata-ipdata-dataprocess"
set COMMON_JARS=%PROJECT_DIR%\target\libs\geodata-common-2.3.3.101.jar
set MAIN_CLASS="com.huawei.wisecloud.location.geodata.common.main.DataAnalysis"
set MAIN_JAR=%PROJECT_DIR%\target\geodata-ipdata-dataprocess-2.3.3.101.jar
set CONFIG_FILE=%PROJECT_DIR%\config\config.properties
cd /d %PROJECT_DIR%
if "%param%"=="debug" (
mvn clean install -DskipTests && spark-submit --driver-memory 10G --num-executors 6 --executor-memory 2G --driver-java-options -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005 --jars %COMMON_JARS% --class %MAIN_CLASS% --master local[4] %MAIN_JAR% %CONFIG_FILE%
) else if "%param%"=="ds" (
spark-submit --driver-memory 10G --num-executors 6 --executor-memory 2G --driver-java-options -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005 --jars %COMMON_JARS% --class %MAIN_CLASS% --master local[4] %MAIN_JAR% %CONFIG_FILE%
) else if "%param%"=="submit" (
spark-submit --driver-memory 10G --num-executors 6 --executor-memory 2G --jars %COMMON_JARS% --class %MAIN_CLASS% --master local[4] %MAIN_JAR% %CONFIG_FILE%
) else (
mvn clean install -DskipTests && spark-submit --driver-memory 10G --num-executors 6 --executor-memory 2G --jars %COMMON_JARS% --class %MAIN_CLASS% --master local[4] %MAIN_JAR% %CONFIG_FILE%
)

三、cmd提交spark任务

提交命令：

ip_spark submit

四、spark任务debug

提交命令：

ip_spark debug

&＃160;

端口在监听，如下图：

&＃160;

&＃160;

&＃160;FAQ:

1、spark任务在提交的时候，出现加载hive的临时数据目录权限失败

&＃160;

&＃160;用命令查询指定文件的权限

命令为：winutils ls software

&＃160;

权限不足的文件如下：

C:\>winutils ls software
FindFileOwnerAndPermission error (1789): ????????????????

&＃160;

权限777的文件如下：

C:\>winutils chmod -R 777 temp

C:\>winutils ls temp
drwxrwxrwx 1 BUILTIN\Administrators NT AUTHORITY\SYSTEM 4096 Dec 8 2020 temp

&＃160;

所以在spark安装的目录下把conf/hive-site.xml文件中的hive的临时数据目录修改为有权限的文件路径

&＃160;