大数据调试环境配置（3）：IDEA外部链接spark调试环境配置

在远程电脑的windows系统上&＃xff0c;部署远程spark代码开发环境&＃xff0c;从而提升效率。

1.CDH5.15.2

2.scala2.11.8

1.新建scala项目

&＃xff08;1&＃xff09;依据模板建立maven项目

&＃xff08;2&＃xff09;输入项目关键名称

&＃xff08;3&＃xff09;选择本地maven仓库对应配置文件settings.xml

&＃xff08;4&＃xff09;填写项目名称&＃xff0c;确认建立项目

&＃xff08;5&＃xff09;修改pom文件中scala对应版本为2.11.8

4.0.0sparktestsparktest1.0-SNAPSHOT20082.11.8scala-tools.orgScala-Tools Maven2 Repositoryhttp://scala-tools.org/repo-releasesscala-tools.orgScala-Tools Maven2 Repositoryhttp://scala-tools.org/repo-releasesorg.scala-langscala-library${scala.version}org.apache.sparkspark-core_${scala.binary}${spark.version}
org.apache.sparkspark-sql_${scala.binary}${spark.version}
org.apache.sparkspark-hive_2.11${spark.version}src/main/scalasrc/test/scalaorg.scala-toolsmaven-scala-plugincompiletestCompile${scala.version}-target:jvm-1.5org.apache.maven.pluginsmaven-eclipse-plugintruech.epfl.lamp.sdt.core.scalabuilderch.epfl.lamp.sdt.core.scalanatureorg.eclipse.jdt.launching.JRE_CONTAINERch.epfl.lamp.sdt.launching.SCALA_CONTAINERorg.scala-toolsmaven-scala-plugin${scala.version}



等待项目相关依赖导入

&＃xff08;6&＃xff09;配置环境变量HADOOP_HOME

否则会报错&＃xff1a;Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries. 

报错原因&＃xff1a;windows下运行spark程序报错&＃xff1a;Failed to locate the winutils binary in the hadoop binary path

&＃xff08;7&＃xff09;添加环境变量Path中对应包含winutils.exe文件的位置

如下图

&＃xff08;8&＃xff09;远程调试&＃xff0c;无法读取hdfs文件&＃xff0c;参考

Hadoop远程调试删除文件报错&＃xff1a;org.apache.hadoop.security.AccessControlException: Permission denied: user&＃61;

&＃xff08;9&＃xff09;重启idea后生效&＃xff0c;可以运行spark程序

2.创建scala代码&＃xff1a;简单wordcount代码

import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}/*** spark Streaming 处理socket数据** 使用nc测试nc -lk 6789*/
object NetworkWordCount {def main(args: Array[String]): Unit &＃61; {val sparkConf&＃61;new SparkConf().setMaster("local[2]").setAppName("NetworkWordCount")/**** 创建StreamingContext需要sparkConf和batch interval*/val ssc&＃61;new StreamingContext(sparkConf,Seconds(5))val lines&＃61;ssc.socketTextStream("bigdata.ibeifeng.com",6789)val result&＃61; lines.flatMap(_.split(" ")).map((_,1)).reduceByKey(_&＃43;_)result.print()ssc.start()ssc.awaitTermination()}
}

3.从CM中下载配置文件hive-site.xml以及log4j.properties文件到resources文件夹下

1.开启spark

2.开启nc端口

nc -lk 6789

3.运行代码即可~