【Spark】用scala2.11编译打包构建镜像
时间:2022-07-22
本文章向大家介绍【Spark】用scala2.11编译打包构建镜像，主要内容包括其使用实例、应用技巧、基本知识点总结和需要注意事项，具有一定的参考价值，需要的朋友可以参考一下。
如果关注 Spark 社区的同学已经知道，从 Spark 3.0 开始，就不再支持用 Scala 2.11 来编译了，Scala 2.11 相关的依赖也被 Owen 去掉了，所以如果要用 Spark 3.0 的同学就需要用 Scala 2.12 去编译打包了。
而如果还在用 Spark 2.x 版本的，至少在 2.4.5 的版本中，已经是将 Scala 2.12 作为默认的 Scala 版本了，如果用户的程序是用 Scala 2.11 写的，也需要将 Spark 用 Scala 2.11 编一次，而 Maven 的仓库里应该只有 Scala 2.12 编出来的 jar 依赖可以供下载，所以如果你的需求是下面那样的：
可以用 Spark 2.4.5，并且用户的代码是用 Scala 2.11 作为依赖的
那么就需要自己用 Scala 2.11 编一次 Spark 的依赖。具体的流程也比较简单，Spark 有提供一个脚本来修改项目 pom.xml 文件关于 Scala 依赖版本的工具，用户尽量不要自己去找 pom.xml 里的 Scala 依赖版本，还挺容易找错找漏的，用脚本 change-scala-version.sh 就可以了。 下面是我从编译打包到构建镜像完整流程的日志，可以供大家参考一下。
➜ spark git:(v2.4.5) dev/change-scala-version.sh --help
Usage: change-scala-version.sh [-h|--help] <version>
where :
-h| --help Display this help text
valid version values : 2.11 2.12
➜ spark git:(v2.4.5) dev/change-scala-version.sh 2.11
dev/../hadoop-cloud/pom.xml
dev/../assembly/pom.xml
dev/../repl/pom.xml
dev/../core/pom.xml
dev/../external/spark-ganglia-lgpl/pom.xml
dev/../external/flume/pom.xml
dev/../external/kafka-0-8/pom.xml
dev/../external/kafka-0-10-sql/pom.xml
dev/../external/kafka-0-8-assembly/pom.xml
dev/../external/kinesis-asl-assembly/pom.xml
dev/../external/flume-assembly/pom.xml
dev/../external/kinesis-asl/pom.xml
dev/../external/docker-integration-tests/pom.xml
dev/../external/kafka-0-10-assembly/pom.xml
dev/../external/kafka-0-10/pom.xml
dev/../external/avro/pom.xml
dev/../external/flume-sink/pom.xml
dev/../tools/pom.xml
dev/../streaming/pom.xml
dev/../mllib-local/pom.xml
dev/../launcher/pom.xml
dev/../pom.xml
dev/../examples/pom.xml
dev/../sql/core/pom.xml
dev/../sql/hive-thriftserver/pom.xml
dev/../sql/catalyst/pom.xml
dev/../sql/hive/pom.xml
dev/../common/network-common/pom.xml
dev/../common/tags/pom.xml
dev/../common/kvstore/pom.xml
dev/../common/network-yarn/pom.xml
dev/../common/unsafe/pom.xml
dev/../common/network-shuffle/pom.xml
dev/../common/sketch/pom.xml
dev/../mllib/pom.xml
dev/../resource-managers/kubernetes/core/pom.xml
dev/../resource-managers/kubernetes/integration-tests/pom.xml
dev/../resource-managers/yarn/pom.xml
dev/../resource-managers/mesos/pom.xml
dev/../graphx/pom.xml
dev/../docs/_plugins/copy_api_dirs.rb
➜ spark git:(v2.4.5) dev/make-distribution.sh --tgz -Phadoop-2.7 -Pkubernetes
+++ dirname dev/make-distribution.sh
++ cd dev/..
++ pwd
+ SPARK_HOME=/root/tx/spark-all/spark
+ DISTDIR=/root/tx/spark-all/spark/dist
+ MAKE_TGZ=false
+ MAKE_PIP=false
+ MAKE_R=false
+ NAME=none
+ MVN=/root/tx/spark-all/spark/build/mvn
+ (( 3 ))
+ case $1 in
+ MAKE_TGZ=true
+ shift
+ (( 2 ))
+ case $1 in
+ break
+ '[' -z /root/txKona-8.0.0-232 ']'
+ '[' -z /root/txKona-8.0.0-232 ']'
++ command -v git
+ '[' /usr/bin/git ']'
++ git rev-parse --short HEAD
+ GITREV=cee4ecb
+ '[' '!' -z cee4ecb ']'
+ GITREVSTRING=' (git revision cee4ecb)'
+ unset GITREV
++ command -v /root/tx/spark-all/spark/build/mvn
+ '[' '!' /root/tx/spark-all/spark/build/mvn ']'
++ /root/tx/spark-all/spark/build/mvn help:evaluate -Dexpression=project.version -Phadoop-2.7 -Pkubernetes
++ grep -v INFO
++ grep -v WARNING
++ tail -n 1
+ VERSION=2.4.5
++ /root/tx/spark-all/spark/build/mvn help:evaluate -Dexpression=scala.binary.version -Phadoop-2.7 -Pkubernetes
++ grep -v INFO
++ grep -v WARNING
++ tail -n 1
+ SCALA_VERSION=2.11
++ /root/tx/spark-all/spark/build/mvn help:evaluate -Dexpression=hadoop.version -Phadoop-2.7 -Pkubernetes
++ grep -v INFO
++ grep -v WARNING
++ tail -n 1
+ SPARK_HADOOP_VERSION=2.7.3
++ /root/tx/spark-all/spark/build/mvn help:evaluate -Dexpression=project.activeProfiles -pl sql/hive -Phadoop-2.7 -Pkubernetes
++ grep -v INFO
++ grep -v WARNING
++ fgrep --count '<id>hive</id>'
++ echo -n
+ SPARK_HIVE=0
+ '[' none == none ']'
+ NAME=2.7.3
+ echo 'Spark version is 2.4.5'
Spark version is 2.4.5
+ '[' true == true ']'
+ echo 'Making spark-2.4.5-bin-2.7.3.tgz'
Making spark-2.4.5-bin-2.7.3.tgz
+ cd /root/tx/spark-all/spark
+ export 'MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=1g'
+ MAVEN_OPTS='-Xmx2g -XX:ReservedCodeCacheSize=1g'
+ BUILD_COMMAND=("$MVN" -T 1C clean package -DskipTests $@)
+ echo -e 'nBuilding with...'
Building with...
+ echo -e '$ /root/tx/spark-all/spark/build/mvn' -T 1C clean package -DskipTests -Phadoop-2.7 '-Pkubernetesn'
$ /root/tx/spark-all/spark/build/mvn -T 1C clean package -DskipTests -Phadoop-2.7 -Pkubernetes
+ /root/tx/spark-all/spark/build/mvn -T 1C clean package -DskipTests -Phadoop-2.7 -Pkubernetes
Using `mvn` from path: /root/apache-maven-3.6.2/bin/mvn
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Spark Project Parent POM [pom]
[INFO] Spark Project Tags [jar]
[INFO] Spark Project Sketch [jar]
[INFO] Spark Project Local DB [jar]
[INFO] Spark Project Networking [jar]
[INFO] Spark Project Shuffle Streaming Service [jar]
[INFO] Spark Project Unsafe [jar]
[INFO] Spark Project Launcher [jar]
[INFO] Spark Project Core [jar]
[INFO] Spark Project ML Local Library [jar]
[INFO] Spark Project GraphX [jar]
[INFO] Spark Project Streaming [jar]
[INFO] Spark Project Catalyst [jar]
[INFO] Spark Project SQL [jar]
[INFO] Spark Project ML Library [jar]
[INFO] Spark Project Tools [jar]
[INFO] Spark Project Hive [jar]
[INFO] Spark Project REPL [jar]
[INFO] Spark Project Kubernetes [jar]
[INFO] Spark Project Assembly [pom]
[INFO] Spark Integration for Kafka 0.10 [jar]
[INFO] Kafka 0.10+ Source for Structured Streaming [jar]
[INFO] Spark Project Examples [jar]
[INFO] Spark Integration for Kafka 0.10 Assembly [jar]
[INFO] Spark Avro [jar]
[INFO]
[INFO] Using the MultiThreadedBuilder implementation with a thread count of 8
[INFO]
.....省略很多
.....省略很多
.....省略很多
.....省略很多
.....省略很多
[INFO] -----------------< org.apache.spark:spark-parent_2.11 >-----------------
[INFO] Building Spark Project Parent POM 2.4.5 [1/25]
[INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork (create-source-jar) @ spark-examples_2.11 ---
[INFO] Building jar: /root/tx/spark-all/spark/examples/target/spark-examples_2.11-2.4.5-test-sources.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 2.4.5:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 47.105 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 27.619 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 4.121 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 13.466 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 16.274 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 3.603 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 13.363 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 7.332 s]
[INFO] Spark Project Core ................................. SUCCESS [03:08 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 20.801 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 17.927 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 34.502 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [01:58 min]
[INFO] Spark Project SQL .................................. SUCCESS [03:38 min]
[INFO] Spark Project ML Library ........................... SUCCESS [02:02 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 26.929 s]
[INFO] Spark Project Hive ................................. SUCCESS [ 50.893 s]
[INFO] Spark Project REPL ................................. SUCCESS [ 4.578 s]
[INFO] Spark Project Kubernetes ........................... SUCCESS [ 16.327 s]
[INFO] Spark Project Assembly ............................. SUCCESS [ 5.431 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 15.950 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 14.692 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 18.164 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 4.847 s]
[INFO] Spark Avro ......................................... SUCCESS [ 10.145 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12:40 min (Wall Clock)
[INFO] Finished at: 2020-07-27T09:11:44+08:00
[INFO] ------------------------------------------------------------------------
+ rm -rf /root/tx/spark-all/spark/dist
+ mkdir -p /root/tx/spark-all/spark/dist/jars
+ echo 'Spark 2.4.5 (git revision cee4ecb) built for Hadoop 2.7.3'
+ echo 'Build flags: -Phadoop-2.7' -Pkubernetes
/*
* 提示：该行代码过长，系统自动注释不进行高亮。一键复制会移除系统注释
* + cp /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/activation-1.1.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/aircompressor-0.10.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/antlr4-runtime-4.7.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/aopalliance-repackaged-2.4.0-b34.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/apacheds-i18n-2.0.0-M15.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/apacheds-kerberos-codec-2.0.0-M15.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/api-asn1-api-1.0.0-M20.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/api-util-1.0.0-M20.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/arpack_combined_all-0.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/arrow-format-0.10.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/arrow-memory-0.10.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/arrow-vector-0.10.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/automaton-1.11-8.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/avro-1.8.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/avro-ipc-1.8.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/avro-mapred-1.8.2-hadoop2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/breeze_2.11-0.13.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/breeze-macros_2.11-0.13.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/chill_2.11-0.9.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/chill-java-0.9.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-beanutils-1.9.4.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-cli-1.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-codec-1.10.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-collections-3.2.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-compiler-3.0.9.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-compress-1.8.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-configuration-1.6.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-crypto-1.0.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-digester-1.8.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-httpclient-3.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-io-2.4.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-lang-2.6.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-lang3-3.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-math3-3.4.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-net-3.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/compress-lzf-1.0.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/core-1.1.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/curator-client-2.7.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/curator-framework-2.7.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/curator-recipes-2.7.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/flatbuffers-1.2.0-3f79e055.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/generex-1.0.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/gson-2.2.4.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/guava-14.0.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-annotations-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-auth-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-client-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-common-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-hdfs-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-app-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-common-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-core-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-jobclient-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-shuffle-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-yarn-api-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-yarn-client-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-yarn-common-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-yarn-server-common-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hk2-api-2.4.0-b34.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hk2-locator-2.4.0-b34.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hk2-utils-2.4.0-b34.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hppc-0.7.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/htrace-core-3.1.0-incubating.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/httpclient-4.5.6.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/httpcore-4.4.10.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/ivy-2.4.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-annotations-2.6.7.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-core-2.6.7.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-core-asl-1.9.13.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-databind-2.6.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-dataformat-yaml-2.6.7.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-jaxrs-1.9.13.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-mapper-asl-1.9.13.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-module-jaxb-annotations-2.6.7.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-module-paranamer-2.7.9.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-module-scala_2.11-2.6.7.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-xc-1.9.13.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/janino-3.0.9.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/javassist-3.18.1-GA.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/javax.annotation-api-1.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/javax.inject-2.4.0-b34.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/javax.servlet-api-3.1.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/javax.ws.rs-api-2.0.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jaxb-api-2.2.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jcl-over-slf4j-1.7.16.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-client-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-common-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-container-servlet-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-container-servlet-core-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-guava-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-media-jaxb-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-server-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jetty-util-6.1.26.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/joda-time-2.9.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/json4s-ast_2.11-3.5.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/json4s-core_2.11-3.5.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/json4s-jackson_2.11-3.5.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/json4s-scalap_2.11-3.5.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jsp-api-2.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jsr305-1.3.9.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jtransforms-2.4.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jul-to-slf4j-1.7.16.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/kryo-shaded-4.0.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/kubernetes-client-4.6.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/kubernetes-model-4.6.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/kubernetes-model-common-4.6.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/leveldbjni-all-1.8.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/log4j-1.2.17.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/logging-interceptor-3.12.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/lz4-java-1.4.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/machinist_2.11-0.6.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/macro-compat_2.11-1.1.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/metrics-core-3.1.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/metrics-graphite-3.1.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/metrics-json-3.1.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/metrics-jvm-3.1.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/minlog-1.3.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/netty-3.9.9.Final.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/netty-all-4.1.42.Final.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/objenesis-2.5.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/okhttp-3.12.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/okio-1.15.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/opencsv-2.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/orc-core-1.5.5-nohive.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/orc-mapreduce-1.5.5-nohive.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/orc-shims-1.5.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/oro-2.0.8.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/osgi-resource-locator-1.0.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/paranamer-2.8.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/parquet-column-1.10.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/parquet-common-1.10.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/parquet-encoding-1.10.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/parquet-format-2.4.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/parquet-hadoop-1.10.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/parquet-jackson-1.10.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/protobuf-java-2.5.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/py4j-0.10.7.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/pyrolite-4.13.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/RoaringBitmap-0.7.45.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/scala-compiler-2.11.12.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/scala-library-2.11.12.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/scala-parser-combinators_2.11-1.1.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/scala-reflect-2.11.12.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/scala-xml_2.11-1.0.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/shapeless_2.11-2.3.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/shims-0.7.45.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/slf4j-api-1.7.16.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/slf4j-log4j12-1.7.16.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/snakeyaml-1.15.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/snappy-java-1.1.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-catalyst_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-core_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-graphx_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-kubernetes_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-kvstore_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-launcher_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-mllib_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-mllib-local_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-network-common_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-network-shuffle_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-repl_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-sketch_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-sql_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-streaming_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-tags_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-tags_2.11-2.4.5-tests.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-unsafe_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spire_2.11-0.13.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spire-macros_2.11-0.13.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/stax-api-1.0-2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/stream-2.7.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/univocity-parsers-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/validation-api-1.1.0.Final.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/xbean-asm6-shaded-4.8.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/xercesImpl-2.9.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/xmlenc-0.52.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/xz-1.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/zjsonpatch-0.3.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/zookeeper-3.4.6.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/zstd-jni-1.3.2-2.jar /root/tx/spark-all/spark/dist/jars/
*/
+ '[' -f '/root/tx/spark-all/spark/common/network-yarn/target/scala*/spark-*-yarn-shuffle.jar' ']'
+ '[' -d /root/tx/spark-all/spark/resource-managers/kubernetes/core/target/ ']'
+ mkdir -p /root/tx/spark-all/spark/dist/kubernetes/
+ cp -a /root/tx/spark-all/spark/resource-managers/kubernetes/docker/src/main/dockerfiles /root/tx/spark-all/spark/dist/kubernetes/
+ cp -a /root/tx/spark-all/spark/resource-managers/kubernetes/integration-tests/tests /root/tx/spark-all/spark/dist/kubernetes/
+ mkdir -p /root/tx/spark-all/spark/dist/examples/jars
+ cp /root/tx/spark-all/spark/examples/target/scala-2.11/jars/aircompressor-0.10.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/commons-codec-1.10.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/commons-lang-2.6.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/orc-core-1.5.5-nohive.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/orc-mapreduce-1.5.5-nohive.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/orc-shims-1.5.5.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/paranamer-2.8.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/scopt_2.11-3.7.0.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/spark-examples_2.11-2.4.5.jar /root/tx/spark-all/spark/dist/examples/jars
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/aircompressor-0.10.jar
+ name=aircompressor-0.10.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/aircompressor-0.10.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/aircompressor-0.10.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/commons-codec-1.10.jar
+ name=commons-codec-1.10.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/commons-codec-1.10.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/commons-codec-1.10.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/commons-lang-2.6.jar
+ name=commons-lang-2.6.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/commons-lang-2.6.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/commons-lang-2.6.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/orc-core-1.5.5-nohive.jar
+ name=orc-core-1.5.5-nohive.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/orc-core-1.5.5-nohive.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/orc-core-1.5.5-nohive.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/orc-mapreduce-1.5.5-nohive.jar
+ name=orc-mapreduce-1.5.5-nohive.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/orc-mapreduce-1.5.5-nohive.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/orc-mapreduce-1.5.5-nohive.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/orc-shims-1.5.5.jar
+ name=orc-shims-1.5.5.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/orc-shims-1.5.5.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/orc-shims-1.5.5.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/paranamer-2.8.jar
+ name=paranamer-2.8.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/paranamer-2.8.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/paranamer-2.8.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/scopt_2.11-3.7.0.jar
+ name=scopt_2.11-3.7.0.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/scopt_2.11-3.7.0.jar ']'
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/spark-examples_2.11-2.4.5.jar
+ name=spark-examples_2.11-2.4.5.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/spark-examples_2.11-2.4.5.jar ']'
+ mkdir -p /root/tx/spark-all/spark/dist/examples/src/main
+ cp -r /root/tx/spark-all/spark/examples/src/main /root/tx/spark-all/spark/dist/examples/src/
+ '[' -e /root/tx/spark-all/spark/LICENSE-binary ']'
+ cp /root/tx/spark-all/spark/LICENSE-binary /root/tx/spark-all/spark/dist/LICENSE
+ cp -r /root/tx/spark-all/spark/licenses-binary /root/tx/spark-all/spark/dist/licenses
+ cp /root/tx/spark-all/spark/NOTICE-binary /root/tx/spark-all/spark/dist/NOTICE
+ '[' -e /root/tx/spark-all/spark/CHANGES.txt ']'
+ cp -r /root/tx/spark-all/spark/data /root/tx/spark-all/spark/dist
+ '[' false == true ']'
+ echo 'Skipping building python distribution package'
Skipping building python distribution package
+ '[' false == true ']'
+ echo 'Skipping building R source package'
Skipping building R source package
+ mkdir /root/tx/spark-all/spark/dist/conf
+ cp /root/tx/spark-all/spark/conf/docker.properties.template /root/tx/spark-all/spark/conf/fairscheduler.xml.template /root/tx/spark-all/spark/conf/log4j.properties.template /root/tx/spark-all/spark/conf/metrics.properties.template /root/tx/spark-all/spark/conf/slaves.template /root/tx/spark-all/spark/conf/spark-defaults.conf.template /root/tx/spark-all/spark/conf/spark-env.sh.template /root/tx/spark-all/spark/dist/conf
+ cp /root/tx/spark-all/spark/README.md /root/tx/spark-all/spark/dist
+ cp -r /root/tx/spark-all/spark/bin /root/tx/spark-all/spark/dist
+ cp -r /root/tx/spark-all/spark/python /root/tx/spark-all/spark/dist
+ '[' false == true ']'
+ cp -r /root/tx/spark-all/spark/sbin /root/tx/spark-all/spark/dist
+ '[' -d /root/tx/spark-all/spark/R/lib/SparkR ']'
+ '[' true == true ']'
+ TARDIR_NAME=spark-2.4.5-bin-2.7.3
+ TARDIR=/root/tx/spark-all/spark/spark-2.4.5-bin-2.7.3
+ rm -rf /root/tx/spark-all/spark/spark-2.4.5-bin-2.7.3
+ cp -r /root/tx/spark-all/spark/dist /root/tx/spark-all/spark/spark-2.4.5-bin-2.7.3
+ tar czf spark-2.4.5-bin-2.7.3.tgz -C /root/tx/spark-all/spark spark-2.4.5-bin-2.7.3
+ rm -rf /root/tx/spark-all/spark/spark-2.4.5-bin-2.7.3
➜ spark git:(v2.4.5) tar zxvf spark-2.4.5-bin-2.7.3.tgz
...省略很多
...省略很多
...省略很多
➜ spark git:(v2.4.5) ✗ cd spark-2.4.5-bin-2.7.3
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ bin/docker-image-tool.sh -r yourhub.com/pulic -t 2.11_2.4.5 build
Sending build context to Docker daemon 201.1MB
Step 1/15 : FROM openjdk:8-jdk-slim
---> a0b0dea106dd
Step 2/15 : ARG spark_jars=jars
---> Running in 788ed25e1956
Removing intermediate container 788ed25e1956
---> db4a6a8ae0d8
Step 3/15 : ARG img_path=kubernetes/dockerfiles
---> Running in fd77ef759bed
Removing intermediate container fd77ef759bed
---> 689207604eb1
Step 4/15 : ARG k8s_tests=kubernetes/tests
---> Running in fe4d6784371b
Removing intermediate container fe4d6784371b
---> 4ac25b12085b
Step 5/15 : RUN set -ex && apt-get update && ln -s /lib /lib64 && apt install -y bash tini libc6 libpam-modules libnss3 && mkdir -p /opt/spark && mkdir -p /opt/spark/work-dir && touch /opt/spark/RELEASE && rm /bin/sh && ln -sv /bin/bash /bin/sh && echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && chgrp root /etc/passwd && chmod ug+rw /etc/passwd && rm -rf /var/cache/apt/*
---> Running in ea60b610c182
+ apt-get update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [121 kB]
Get:3 http://security.debian.org/debian-security buster/updates/main amd64 Packages [213 kB]
Get:4 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7905 kB]
^C
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ vi kubernetes/dockerfiles/spark/Dockerfile
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ bin/docker-image-tool.sh -r yourhub.com/pulic -t 2.11_2.4.5 build
Sending build context to Docker daemon 201.1MB
Step 1/15 : FROM openjdk:8-jdk-slim
---> a0b0dea106dd
Step 2/15 : ARG spark_jars=jars
---> Using cache
---> db4a6a8ae0d8
Step 3/15 : ARG img_path=kubernetes/dockerfiles
---> Using cache
---> 689207604eb1
Step 4/15 : ARG k8s_tests=kubernetes/tests
---> Using cache
---> 4ac25b12085b
Step 5/15 : RUN set -ex && apt-get update && ln -s /lib /lib64 && apt install -y bash tini libc6 libpam-modules libnss3 && mkdir -p /opt/spark && mkdir -p /opt/spark/work-dir && touch /opt/spark/RELEASE && rm /bin/sh && ln -sv /bin/bash /bin/sh && echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && chgrp root /etc/passwd && chmod ug+rw /etc/passwd && rm -rf /var/cache/apt/*
---> Running in 3e9d1d5ac7c1
+ apt-get update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [121 kB]
Get:3 http://security.debian.org/debian-security buster/updates/main amd64 Packages [213 kB]
Get:4 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7905 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [7868 B]
Fetched 8365 kB in 3s (3140 kB/s)
Reading package lists...
+ ln -s /lib /lib64
+ apt install -y bash tini libc6 libpam-modules libnss3
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Reading package lists...
Building dependency tree...
Reading state information...
bash is already the newest version (5.0-4).
bash set to manually installed.
libc6 is already the newest version (2.28-10).
libc6 set to manually installed.
libpam-modules is already the newest version (1.3.1-5).
libpam-modules set to manually installed.
The following package was automatically installed and is no longer required:
lsb-base
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
libnspr4 libsqlite3-0
The following NEW packages will be installed:
libnspr4 libnss3 libsqlite3-0 tini
0 upgraded, 4 newly installed, 0 to remove and 15 not upgraded.
Need to get 2159 kB of archives.
After this operation, 6175 kB of additional disk space will be used.
Get:1 http://security.debian.org/debian-security buster/updates/main amd64 libnss3 amd64 2:3.42.1-1+deb10u3 [1159 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libnspr4 amd64 2:4.20-1 [112 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 libsqlite3-0 amd64 3.27.2-3 [641 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 tini amd64 0.18.0-1 [247 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 2159 kB in 1s (3301 kB/s)
Selecting previously unselected package libnspr4:amd64.
(Reading database ... 6888 files and directories currently installed.)
Preparing to unpack .../libnspr4_2%3a4.20-1_amd64.deb ...
Unpacking libnspr4:amd64 (2:4.20-1) ...
Selecting previously unselected package libsqlite3-0:amd64.
Preparing to unpack .../libsqlite3-0_3.27.2-3_amd64.deb ...
Unpacking libsqlite3-0:amd64 (3.27.2-3) ...
Selecting previously unselected package libnss3:amd64.
Preparing to unpack .../libnss3_2%3a3.42.1-1+deb10u3_amd64.deb ...
Unpacking libnss3:amd64 (2:3.42.1-1+deb10u3) ...
Selecting previously unselected package tini.
Preparing to unpack .../tini_0.18.0-1_amd64.deb ...
Unpacking tini (0.18.0-1) ...
Setting up libsqlite3-0:amd64 (3.27.2-3) ...
Setting up libnspr4:amd64 (2:4.20-1) ...
Setting up tini (0.18.0-1) ...
Setting up libnss3:amd64 (2:3.42.1-1+deb10u3) ...
Processing triggers for libc-bin (2.28-10) ...
+ mkdir -p /opt/spark
+ mkdir -p /opt/spark/work-dir
+ touch /opt/spark/RELEASE
+ rm /bin/sh
+ ln -sv /bin/bash /bin/sh
'/bin/sh' -> '/bin/bash'
+ echo auth required pam_wheel.so use_uid
+ chgrp root /etc/passwd
+ chmod ug+rw /etc/passwd
+ rm -rf /var/cache/apt/archives
Removing intermediate container 3e9d1d5ac7c1
---> 1ffbc3c2f99f
Step 6/15 : COPY ${spark_jars} /opt/spark/jars
---> 9a4610c4c102
Step 7/15 : COPY bin /opt/spark/bin
---> 06c908997a9f
Step 8/15 : COPY sbin /opt/spark/sbin
---> b32be7e7dda4
Step 9/15 : COPY ${img_path}/spark/entrypoint.sh /opt/
---> 6bee90987e7f
Step 10/15 : COPY examples /opt/spark/examples
---> 65775aca9747
Step 11/15 : COPY ${k8s_tests} /opt/spark/tests
---> 6768a8d5c69a
Step 12/15 : COPY data /opt/spark/data
---> 1520647faf15
Step 13/15 : ENV SPARK_HOME /opt/spark
---> Running in 7688890172f1
Removing intermediate container 7688890172f1
---> bf2dd6442c38
Step 14/15 : WORKDIR /opt/spark/work-dir
---> Running in b42600a07aaf
Removing intermediate container b42600a07aaf
---> 96c18bdcae55
Step 15/15 : ENTRYPOINT [ "/opt/entrypoint.sh" ]
---> Running in 62490f481000
Removing intermediate container 62490f481000
---> c016374ced2b
Successfully built c016374ced2b
Successfully tagged yourhub.com/pulic/spark:2.11_2.4.5
Sending build context to Docker daemon 201.1MB
Step 1/9 : ARG base_img
Step 2/9 : FROM $base_img
---> c016374ced2b
Step 3/9 : WORKDIR /
---> Running in ebdd964393d8
Removing intermediate container ebdd964393d8
---> 76ca1b8a91e3
Step 4/9 : RUN mkdir ${SPARK_HOME}/python
---> Running in 3073d6d8a3f7
Removing intermediate container 3073d6d8a3f7
---> 176a7fb978a3
Step 5/9 : RUN apt install -y python python-pip && apt install -y python3 python3-pip && rm -r /usr/lib/python*/ensurepip && pip install --upgrade pip setuptools && rm -r /root/.cache && rm -rf /var/cache/apt/*
---> Running in 4a5fdb7337fa
...
...
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ ls
bin data jars LICENSE NOTICE README.md sbin
conf examples kubernetes licenses python RELEASE
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ docker images | grep -i "2.11_2.4.5"
yourhub.com/pulic/spark-py 2.11_2.4.5 c7190686b340 About a minute ago 1GB
yourhub.com/pulic/spark 2.11_2.4.5 c016374ced2b 2 minutes ago 504MB
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ docker push yourhub.com/pulic/spark:2.11_2.4.5
The push refers to repository [yourhub.com/pulic/spark]
2.11_2.4.5: digest: sha256:60bccd043dd4d16c7cc2b0879c6de394d1d48561f624082ddd627af2f9c96084 size: 2838
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ docker push yourhub.com/pulic/spark-py:2.11_2.4.5
The push refers to repository [yourhub.com/pulic/spark-py]
2.11_2.4.5: digest: sha256:49c0b17c8addc21bd30735ccd6cdc88c3c57e826f1b138cbedd1b13fa23258f8 size: 3469
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ docker run -it yourhub.com/pulic/spark:2.11_2.4.5 bash
++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/bash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/bash ']'
+ SPARK_K8S_CMD=bash
+ case "$SPARK_K8S_CMD" in
+ echo 'Non-spark-on-k8s command provided, proceeding in pass-through mode...'
Non-spark-on-k8s command provided, proceeding in pass-through mode...
+ exec /usr/bin/tini -s -- bash
root@a3f20a0deb94:/opt/spark/work-dir#
root@a3f20a0deb94:/opt/spark/work-dir# ls
root@a3f20a0deb94:/opt/spark/work-dir# ls
root@a3f20a0deb94:/opt/spark/work-dir# cd ..
root@a3f20a0deb94:/opt/spark# ls
RELEASE bin data examples jars sbin tests work-dir
root@a3f20a0deb94:/opt/spark# ls jars/
RoaringBitmap-0.7.45.jar hk2-locator-2.4.0-b34.jar objenesis-2.5.1.jar
activation-1.1.1.jar hk2-utils-2.4.0-b34.jar okhttp-3.12.0.jar
aircompressor-0.10.jar hppc-0.7.2.jar okio-1.15.0.jar
antlr4-runtime-4.7.jar htrace-core-3.1.0-incubating.jar opencsv-2.3.jar
aopalliance-repackaged-2.4.0-b34.jar httpclient-4.5.6.jar orc-core-1.5.5-nohive.jar
apacheds-i18n-2.0.0-M15.jar httpcore-4.4.10.jar orc-mapreduce-1.5.5-nohive.jar
apacheds-kerberos-codec-2.0.0-M15.jar ivy-2.4.0.jar orc-shims-1.5.5.jar
api-asn1-api-1.0.0-M20.jar jackson-annotations-2.6.7.jar oro-2.0.8.jar
api-util-1.0.0-M20.jar jackson-core-2.6.7.jar osgi-resource-locator-1.0.1.jar
arpack_combined_all-0.1.jar jackson-core-asl-1.9.13.jar paranamer-2.8.jar
arrow-format-0.10.0.jar jackson-databind-2.6.7.3.jar parquet-column-1.10.1.jar
arrow-memory-0.10.0.jar jackson-dataformat-yaml-2.6.7.jar parquet-common-1.10.1.jar
arrow-vector-0.10.0.jar jackson-jaxrs-1.9.13.jar parquet-encoding-1.10.1.jar
automaton-1.11-8.jar jackson-mapper-asl-1.9.13.jar parquet-format-2.4.0.jar
avro-1.8.2.jar jackson-module-jaxb-annotations-2.6.7.jar parquet-hadoop-1.10.1.jar
avro-ipc-1.8.2.jar jackson-module-paranamer-2.7.9.jar parquet-jackson-1.10.1.jar
avro-mapred-1.8.2-hadoop2.jar jackson-module-scala_2.11-2.6.7.1.jar protobuf-java-2.5.0.jar
breeze-macros_2.11-0.13.2.jar jackson-xc-1.9.13.jar py4j-0.10.7.jar
breeze_2.11-0.13.2.jar janino-3.0.9.jar pyrolite-4.13.jar
chill-java-0.9.3.jar javassist-3.18.1-GA.jar scala-compiler-2.11.12.jar
chill_2.11-0.9.3.jar javax.annotation-api-1.2.jar scala-library-2.11.12.jar
commons-beanutils-1.9.4.jar javax.inject-2.4.0-b34.jar scala-parser-combinators_2.11-1.1.0.jar
commons-cli-1.2.jar javax.servlet-api-3.1.0.jar scala-reflect-2.11.12.jar
commons-codec-1.10.jar javax.ws.rs-api-2.0.1.jar scala-xml_2.11-1.0.5.jar
commons-collections-3.2.2.jar jaxb-api-2.2.2.jar shapeless_2.11-2.3.2.jar
commons-compiler-3.0.9.jar jcl-over-slf4j-1.7.16.jar shims-0.7.45.jar
commons-compress-1.8.1.jar jersey-client-2.22.2.jar slf4j-api-1.7.16.jar
commons-configuration-1.6.jar jersey-common-2.22.2.jar slf4j-log4j12-1.7.16.jar
commons-crypto-1.0.0.jar jersey-container-servlet-2.22.2.jar snakeyaml-1.15.jar
commons-digester-1.8.jar jersey-container-servlet-core-2.22.2.jar snappy-java-1.1.7.3.jar
commons-httpclient-3.1.jar jersey-guava-2.22.2.jar spark-catalyst_2.11-2.4.5.jar
commons-io-2.4.jar jersey-media-jaxb-2.22.2.jar spark-core_2.11-2.4.5.jar
commons-lang-2.6.jar jersey-server-2.22.2.jar spark-graphx_2.11-2.4.5.jar
commons-lang3-3.5.jar jetty-util-6.1.26.jar spark-kubernetes_2.11-2.4.5.jar
commons-math3-3.4.1.jar joda-time-2.9.3.jar spark-kvstore_2.11-2.4.5.jar
commons-net-3.1.jar json4s-ast_2.11-3.5.3.jar spark-launcher_2.11-2.4.5.jar
compress-lzf-1.0.3.jar json4s-core_2.11-3.5.3.jar spark-mllib-local_2.11-2.4.5.jar
core-1.1.2.jar json4s-jackson_2.11-3.5.3.jar spark-mllib_2.11-2.4.5.jar
curator-client-2.7.1.jar json4s-scalap_2.11-3.5.3.jar spark-network-common_2.11-2.4.5.jar
curator-framework-2.7.1.jar jsp-api-2.1.jar spark-network-shuffle_2.11-2.4.5.jar
curator-recipes-2.7.1.jar jsr305-1.3.9.jar spark-repl_2.11-2.4.5.jar
flatbuffers-1.2.0-3f79e055.jar jtransforms-2.4.0.jar spark-sketch_2.11-2.4.5.jar
generex-1.0.2.jar jul-to-slf4j-1.7.16.jar spark-sql_2.11-2.4.5.jar
gson-2.2.4.jar kryo-shaded-4.0.2.jar spark-streaming_2.11-2.4.5.jar
guava-14.0.1.jar kubernetes-client-4.6.1.jar spark-tags_2.11-2.4.5-tests.jar
hadoop-annotations-2.7.3.jar kubernetes-model-4.6.1.jar spark-tags_2.11-2.4.5.jar
hadoop-auth-2.7.3.jar kubernetes-model-common-4.6.1.jar spark-unsafe_2.11-2.4.5.jar
hadoop-client-2.7.3.jar leveldbjni-all-1.8.jar spire-macros_2.11-0.13.0.jar
hadoop-common-2.7.3.jar log4j-1.2.17.jar spire_2.11-0.13.0.jar
hadoop-hdfs-2.7.3.jar logging-interceptor-3.12.0.jar stax-api-1.0-2.jar
hadoop-mapreduce-client-app-2.7.3.jar lz4-java-1.4.0.jar stream-2.7.0.jar
hadoop-mapreduce-client-common-2.7.3.jar machinist_2.11-0.6.1.jar univocity-parsers-2.7.3.jar
hadoop-mapreduce-client-core-2.7.3.jar macro-compat_2.11-1.1.1.jar validation-api-1.1.0.Final.jar
hadoop-mapreduce-client-jobclient-2.7.3.jar metrics-core-3.1.5.jar xbean-asm6-shaded-4.8.jar
hadoop-mapreduce-client-shuffle-2.7.3.jar metrics-graphite-3.1.5.jar xercesImpl-2.9.1.jar
hadoop-yarn-api-2.7.3.jar metrics-json-3.1.5.jar xmlenc-0.52.jar
hadoop-yarn-client-2.7.3.jar metrics-jvm-3.1.5.jar xz-1.5.jar
hadoop-yarn-common-2.7.3.jar minlog-1.3.0.jar zjsonpatch-0.3.0.jar
hadoop-yarn-server-common-2.7.3.jar netty-3.9.9.Final.jar zookeeper-3.4.6.jar
hk2-api-2.4.0-b34.jar
- 任意文件下载引发的思考
- LSTM入门详解
- 如何将CDH集群JAVA升级至JDK8
- 如何将Kerberos环境下CDH集群JAVA升级至JDK8
- 干货｜如何做准确率达98%的交通标志识别系统？
- 用57行代码搞定花8000万美元采购车牌识别项目
- Cloudera Manager Server服务在RedHat7状态显示异常分析
- 开源 | 基于Python的人脸识别：识别准确率高达99.38%！
- 转录组数据的基因表达变化情况探索
- 如何配置Kerberos服务的高可用
- 利用深度学习生成梵高风格画像
- 使用Python-Requests实现ODL对OVS的流表下发
- Keras入门必看教程
- bedtools 用法大全(一文就够吧)
- JavaScript 教程
- JavaScript 编辑工具
- JavaScript 与HTML
- JavaScript 与Java
- JavaScript 数据结构
- JavaScript 基本数据类型
- JavaScript 特殊数据类型
- JavaScript 运算符
- JavaScript typeof 运算符
- JavaScript 表达式
- JavaScript 类型转换
- JavaScript 基本语法
- JavaScript 注释
- Javascript 基本处理流程
- Javascript 选择结构
- Javascript if 语句
- Javascript if 语句的嵌套
- Javascript switch 语句
- Javascript 循环结构
- Javascript 循环结构实例
- Javascript 跳转语句
- Javascript 控制语句总结
- Javascript 函数介绍
- Javascript 函数的定义
- Javascript 函数调用
- Javascript 几种特殊的函数
- JavaScript 内置函数简介
- Javascript eval() 函数
- Javascript isFinite() 函数
- Javascript isNaN() 函数
- parseInt() 与 parseFloat()
- escape() 与 unescape()
- Javascript 字符串介绍
- Javascript length属性
- javascript 字符串函数
- Javascript 日期对象简介
- Javascript 日期对象用途
- Date 对象属性和方法
- Javascript 数组是什么
- Javascript 创建数组
- Javascript 数组赋值与取值
- Javascript 数组属性和方法
- Day15.异常的处理
- 「翻译」在生物信息学中使用 GNU-Parallel
- Kafka运维小贴士 | Kafka 消息监控
- MySQL慢查询优化 | 联结原理
- MySQL千万大表优化实践
- GitHub 标星过万！计算机与网络知识总结电子书下载！
- 盘一盘，那些提效/创意的 vscode 插件
- 基于jenkins实现手动拉取码云代码，实现半自动化部署
- so easy！网页骨架屏自动生成方案（dps）
- 深夜，我偷听到程序员要对session下手……
- CAM 系列论文阅读总结
- kafka客户端指标上报Prometheus方案(已开源)
- 2020-09-25：rust中Point是结构体类型，【let p1=Point{x:25,y:25}；let p2=p1；】...
- R语言在BRFSS数据中可视化分析探索糖尿病的影响因素
- R语言可视化探索BRFSS数据并逻辑回归Logistic回归预测中风