spark编译：构建基于hadoop的spark安装包及遇到问题总结

问题导读 1.spark集群能否单独运行？ 2.如何在spark中指定想编译的hadoop版本？ 3.构建的时候，版本选择需要注意什么？上一篇如何查看spark与hadoop、kafka、Scala、flume、hive等兼容版本【适用于任何版本】 http://www.aboutyun.com/forum.php?mod=viewthread&tid=23252 构建基于hadoop的spark安装包

上一篇说了spark与其它组件兼容的版本，这里具体说下如何构建基于hadoop的spark安装包。首先我们需要有spark源码，上一篇已经交给大家如何使用git下载。当然我这里提供了百度网盘链接链接：http://pan.baidu.com/s/1gfMpTqb 密码：c6dc 默认情况下，spark的执行不需要hadoop，也就是说没有hadoop集群下，spark集群可以设置，启动，使用。。尽管如此，如果你使用spark如Yarn的执行模式，或则访问hdfs创建rdd，它将会依赖hadoop。如果是这种情况，你的spark安装包必须兼容你所使用的hadoop集群的安装包如果你使用的是spark2.3.0对应的hadoop默认为2.6.在假如使用的是spark1.2.0对应的是hadoop2.4. 这里需要注意的是：有些hadoop版本，是有小版本的，比如hadoop2.6.5，hadoop2.7有hadoop2.7.1，hadoop2.7.3.对于hadoop版本的+或则-的小版本之间，它们与spark大多都是兼容可以正常运行的。 spark构建运行开发者指定hadoop版本，你可以直接使用maven命令或则 make-distribution.sh脚本生成安装包。例如构建spark2.3.0，hadoop2.6 maven profile 使用-P选项, 如: $SPARK_SRC/make-distribution.sh --tgz -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6 -Phive spark2.3.0支持2.6，2.7 如何自定义hadoop版本基本的你需要在pom文件中添加profile指定hadoop版本假如你想构建 Hadoop 2.6.5，按照下面步骤第一步：在 $SPARK_SRC/pom.xml中添加maven profile hadoop-2.6.5 在<profiles> 部分

[XML] 纯文本查看复制代码

<profile>
    <id>hadoop-2.6.5</id>
    <properties>
        <hadoop.version>2.6.5</hadoop.version>
        <protobuf.version>2.5.0</protobuf.version>
        <jets3t.version>0.9.0</jets3t.version>
        <commons.math3.version>3.1.1</commons.math3.version>
        <avro.mapred.classifier>hadoop2</avro.mapred.classifier>
    </properties>
</profile>

protobuf, jets3t, commons.math3 和avro.mapred.classifier 版本都是来自Apache Hadoop 2.5.2 二进制分布式libs. 你需要修改要构建的hadoop的对应版本。

对于这个avro.mapred.classifier，大家可以找找，不过在spark2.3.0 pom文件中也是有的 https://github.com/apache/spark/blob/master/pom.xml

上面是比较复杂的，我们也可以使用最简单的方式：

[XML] 纯文本查看复制代码

<profile>
    <id>hadoop-2.6.5</id>
</profile>

第二步：构建执行使用make-distribution.sh构建

[Bash shell] 纯文本查看复制代码

$SPARK_SRC/make-distribution.sh --tgz -Pyarn -Phadoop-2.6.5 -Dhadoop.version=2.6.5 -Phive

使用maven命令构建

[Bash shell] 纯文本查看复制代码

mvn -Pyarn -Dhadoop.version=2.6.5 -Dyarn.version=2.6.5 -DskipTests clean package

构建成功后，可以找到spark-2.*.*-bin-2.6.5.tgz 注意：这种方式建议使用在hadoop小版本，对于hadoop主版本即使构建成功，也可能在生产中遇到一些问题。参考： https://etushar89.wordpress.com/2014/12/28/building-apache-spark-against-specific-hadoop-version/

构建基于hadoop的spark安装包实践及遇到问题总结

spark编译时间还是比较长的，可能要一两个小时，而且有时候卡住那不动。在编译的过程中，有编译失败，也有编译成功的。总之两个条件： 1.跟版本有关系版本不对，可能编译失败 2.跟网速有关系网速不好，有些下载失败，导致编译失败。使用hadoop2.6.5

[Bash shell] 纯文本查看复制代码

$SPARK_SRC/make-distribution.sh --tgz -Pyarn -Phadoop-2.6.5 -Dhadoop.version=2.6.5 -Phive

这里使用的是spark2.3.0，hadoop版本为2.6.5，最后编译失败。报错如下

[Bash shell] 纯文本查看复制代码

[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Skipping Spark Integration for Kafka 0.10 Assembly
[INFO] This project has been banned from the build due to previous failures.
[INFO] ------------------------------------------------------------------------
[INFO] Spark Project Parent POM ........................... SUCCESS [08:36 min]
[INFO] Spark Project Tags ................................. SUCCESS [04:56 min]
[INFO] Spark Project Sketch ............................... SUCCESS [ 10.413 s]
[INFO] Spark Project Local DB ............................. SUCCESS [01:15 min]
[INFO] Spark Project Networking ........................... SUCCESS [ 36.812 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 11.964 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 34.261 s]
[INFO] Spark Project Launcher ............................. SUCCESS [02:40 min]
[INFO] Spark Project Core ................................. SUCCESS [12:34 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [02:19 min]
[INFO] Spark Project GraphX ............................... SUCCESS [01:29 min]
[INFO] Spark Project Streaming ............................ SUCCESS [01:52 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [05:25 min]
[INFO] Spark Project SQL .................................. FAILURE [10:13 min]
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SUCCESS [ 17.955 s]
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 19.981 s]
[INFO] Spark Project YARN ................................. SUCCESS [01:14 min]
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [01:33 min]
[INFO] Kafka 0.10 Source for Structured Streaming ......... SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 56:26 min
[INFO] Finished at: 2017-11-08T10:44:58+08:00
[INFO] Final Memory: 65M/296M
[INFO] ------------------------------------------------------------------------
[WARNING] The requested profile "hadoop-2.6.5" could not be activated because it does not exist.
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) on project spark-sql_2.11: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile failed. CompileFailed -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :spark-sql_2.11

有一个错误和警告是比较关键的 [WARNING] The requested profile "hadoop-2.6.5" could not be activated because it does not exist. [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) on project spark-sql_2.11: 上面是profile中是没有hadoop-2.6.5，因此我们需要增加profile

[XML] 纯文本查看复制代码

<profile>
    <id>hadoop-2.6.5</id>
</profile>

第二个问题不能执行，这个可能就跟网速有关系。多次执行仍然失败，也可能跟版本有关系更换hadoop2.7.1 上面失败，接着我们尝试hadoop2.7.1

[XML] 纯文本查看复制代码

<profile>
    <id>hadoop-2.6.5</id>
</profile>

编译成功但是报错如下

[Bash shell] 纯文本查看复制代码

+ TARDIR_NAME='spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.'
+ TARDIR='/home/aboutyun/spark/spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.'
+ rm -rf '/home/aboutyun/spark/spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.'
+ cp -r /home/aboutyun/spark/dist '/home/aboutyun/spark/spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.'
+ tar czf 'spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist..tgz' -C /home/aboutyun/spark 'spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.'
+ rm -rf '/home/aboutyun/spark/spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.'

[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.这句很关键，意思是虽然编译成功，但是不能使用。在后面将截图给大家看。所以在pom.xml文件中添加如下属性

[Bash shell] 纯文本查看复制代码

[mw_shl_code=bash,true]<profile>
<id>hadoop-2.7.1</id>
</profile>

[/mw_shl_code] 编译成功

下面我们通过winscp查看第一个为未添加profile，第二个添加后，编译成功。