作者:朱泊潇 | 来源:互联网 | 2023-09-25 11:13
1.下载源码 其他版本下载(当前页面拉到底部看到下图信息): 选择需要下载的版本: 选择版本进入文件夹,比如:下载2.3.1版本源码: 2.环境配置需要配置环境变量的程序:JDK1
1. 下载源码
![Spark源码编译(windows) Spark源码编译(windows)](/default/index/img?u=L2RlZmF1bHQvaW5kZXgvaW1nP3U9YUhSMGNITTZMeTl3YVdGdWMyaGxiaTVqYjIwdmFXMWhaMlZ6THpJNE9DODRObVZrWmpZME5XVmhaVGxtTmpFNFpXWTNabVU1TmpjeE1tRm1OekZqTUM1d2JtYz0=)
其他版本下载(当前页面拉到底部看到下图信息):
![Spark源码编译(windows) Spark源码编译(windows)](/default/index/img?u=L2RlZmF1bHQvaW5kZXgvaW1nP3U9YUhSMGNITTZMeTl3YVdGdWMyaGxiaTVqYjIwdmFXMWhaMlZ6THpreU5pODVOMkkwWVdKbE5tWTFObU5sTWpoa1lUUmlOalkzTXpRMlpqWTFOMlZpWlM1d2JtYz0=)
选择需要下载的版本:
![Spark源码编译(windows) Spark源码编译(windows)](/default/index/img?u=L2RlZmF1bHQvaW5kZXgvaW1nP3U9YUhSMGNITTZMeTl3YVdGdWMyaGxiaTVqYjIwdmFXMWhaMlZ6THprMU55OWpZMk16TURObE1Ua3paVEEzTkdJMVlUQXdORGsyTVdOalpEazBZelUxTlM1d2JtYz0=)
选择版本进入文件夹,比如:下载2.3.1版本源码:
![Spark源码编译(windows) Spark源码编译(windows)](/default/index/img?u=L2RlZmF1bHQvaW5kZXgvaW1nP3U9YUhSMGNITTZMeTl3YVdGdWMyaGxiaTVqYjIwdmFXMWhaMlZ6THpreU1pOWxOMk14TjJVd1kyVmpZekZsWXpVNVpXTmhOR1l6WW1NMk9UaGxNelJsTWk1d2JtYz0=)
2. 环境配置
需要配置环境变量的程序:
JDK1.8(注意JDK版本,具体参考spark版本)
Git
Maven (这里采用maven编译源码,注意版本)
安装完上面三个程序要配置环境变量。
JDK1.8环境配置参考 : https://jingyan.baidu.com/article/6dad5075d1dc40a123e36ea3.html
Maven环境变量配置参考:https://jingyan.baidu.com/article/ce09321ba734f02bff858fcf.html
Git环境变量配置:
在环境变量中创建:变量名:GIT_HOME 变量值:D:\Program Files\Git (自己Git安装目录)
在Path中添加: ;%GIT_HOME%\bin (编译时会用到Git的bash)
3. 编译
3.1 先看官网介绍
![Spark源码编译(windows) Spark源码编译(windows)](/default/index/img?u=L2RlZmF1bHQvaW5kZXgvaW1nP3U9YUhSMGNITTZMeTl3YVdGdWMyaGxiaTVqYjIwdmFXMWhaMlZ6THpJeU1TOHlNR1UyWm1SaE16VTJaR0ZpTnprMU16ZG1OVFF3TVdOak9UTTNPV0UxTlM1d2JtYz0=)
笔记:
1) 注意Maven版本和JDK版本;
2)配置MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m";
建议MAVEN_OPTS配置到环境变量中,在变量中新添加:MAVEN_OPTS 值:-Xmx2g -XX:ReservedCodeCacheSize=512m
::
![Spark源码编译(windows) Spark源码编译(windows)](/default/index/img?u=L2RlZmF1bHQvaW5kZXgvaW1nP3U9YUhSMGNITTZMeTl3YVdGdWMyaGxiaTVqYjIwdmFXMWhaMlZ6THpVMU5pODJNbUkzTWpSaU1tUTRaVEExT0RsaVltRXpPVFZrWWpVeVkyUTVPV1k1WXk1d2JtYz0=)
3.2 编译演示
这里以编译spark_streaming为例:
1). 进入\spark-2.3.1\streaming 打开cmd 或者从cmd窗口进入\spark-2.3.1\streaming目录;
2). 运行命令: mvn -DskipTests clean package
![Spark源码编译(windows) Spark源码编译(windows)](/default/index/img?u=L2RlZmF1bHQvaW5kZXgvaW1nP3U9YUhSMGNITTZMeTl3YVdGdWMyaGxiaTVqYjIwdmFXMWhaMlZ6THpjMk1TODBaakV3WmpFeFpHWTRNVEl5TVdJMU1qbGxPVFJqT0RVMU9UTmlaakpsT1M1d2JtYz0=)
3). 编译成功
![Spark源码编译(windows) Spark源码编译(windows)](/default/index/img?u=L2RlZmF1bHQvaW5kZXgvaW1nP3U9YUhSMGNITTZMeTl3YVdGdWMyaGxiaTVqYjIwdmFXMWhaMlZ6THprME1DOWpOMkpoWkdFeFpEZG1OVEZoT1RBM1pqWXlPR0ZpWVRkak9EYzROamc1WXk1d2JtYz0=)
4)查看结果
在spark streaming目录下生成了target 文件夹,进入目录可以看到:
![Spark源码编译(windows) Spark源码编译(windows)](/default/index/img?u=L2RlZmF1bHQvaW5kZXgvaW1nP3U9YUhSMGNITTZMeTl3YVdGdWMyaGxiaTVqYjIwdmFXMWhaMlZ6THpVME5pOWpZVFExTlRoa01XWmxOelEzTUdabFptUmtZakppTW1Vek1UVm1ORFl3TWk1d2JtYz0=)
如果需要编译Spark所有组件代码,在spark-x.x.x目录下,按上面步骤以此操作。