IDEA配置spark与pycharm配置spark教程

eclipse配置spark1.6.0教程
https://kevin12.iteye.com/blog/2274179

这里注意修改,根据自己的修改
com.Johnson
Spark_2.3.1
1.0-SNAPSHOT

4.0.0

com.Johnson
Spark_2.3.1
1.0-SNAPSHOTUTF-82.11.82.3.12.6.01.2.04.14.0
scala-tools.orgScala-Tools Maven2 Repositoryhttp://scala-tools.org/repo-releasesclouderahttps://repository.cloudera.com/artifactory/cloudera-repos/scala-tools.orgScala-Tools Maven2 Repositoryhttp://scala-tools.org/repo-releases
org.scala-langscala-library${scala.version}junitjunit4.4testorg.specsspecs1.2.5testjavax.servletjavax.servlet-api3.0.1com.fasterxml.jackson.corejackson-databind2.4.4com.fasterxml.jackson.corejackson-core2.6.7com.fasterxml.jackson.corejackson-databind2.6.7com.fasterxml.jackson.modulejackson-module-scala_2.112.6.7.1com.fasterxml.jackson.modulejackson-module-paranamer2.7.9com.fasterxml.jackson.corejackson-annotations2.6.7org.apache.logging.log4jlog4j-core2.8.2org.apache.hbasehbase-client${hbase.version}org.apache.hbasehbase-common${hbase.version}org.apache.hadoophadoop-hdfs${hadoop.version}org.apache.hadoophadoop-common${hadoop.version}org.apache.hbasehbase-server${hbase.version}org.apache.zookeeperzookeeper3.4.5com.101teczkclient0.10com.yammer.metricsmetrics-core2.2.0org.apache.hbasehbase-protocol${hbase.version}org.apache.htracehtrace-core3.2.0-incubatingio.nettynetty-all4.1.24.Finalorg.threetenthreeten-extra1.2com.cloudera.sparktssparkts0.4.1mysqlmysql-connector-java5.1.38org.apache.sparkspark-streaming_2.11${spark.version}org.apache.sparkspark-mllib_2.11${spark.version}org.apache.hadoophadoop-client${hadoop.version}org.apache.sparkspark-streaming-kafka-0-10_2.11${spark.version}org.apache.kafkakafka-clients0.10.2.0net.jpountz.lz4lz4org.apache.kafkakafka_2.110.10.2.0org.apache.sparkspark-core_2.11${spark.version}org.apache.sparkspark-sql_2.11${spark.version}org.apache.sparkspark-hive_2.11${spark.version}org.apache.sparkspark-catalyst_2.11${spark.version}org.apache.hadoophadoop-mapreduce-client-core${hadoop.version}org.apache.hadoophadoop-mapreduce-client-common${hadoop.version}commons-clicommons-cli1.2src/main/scalasrc/test/scalaorg.scala-toolsmaven-scala-plugincompiletestCompile${scala.version}-target:jvm-1.8org.apache.maven.pluginsmaven-eclipse-plugintruech.epfl.lamp.sdt.core.scalabuilderch.epfl.lamp.sdt.core.scalanatureorg.eclipse.jdt.launching.JRE_CONTAINERch.epfl.lamp.sdt.launching.SCALA_CONTAINER

org.scala-toolsmaven-scala-plugin${scala.version}



Pycharm配置spark
1&＃xff1a;注意修改自己的路径

或者将pyspark复制到anaconda的site-packages目录下&＃xff1a;

2&＃xff1a;

SPARK_HOME ***
PYTHONPATH ***