如何使用Hue创建Spark1和Spark2的Oozie工作流

时间:2022-05-06
本文章向大家介绍如何使用Hue创建Spark1和Spark2的Oozie工作流，主要内容包括其使用实例、应用技巧、基本知识点总结和需要注意事项，具有一定的参考价值，需要的朋友可以参考一下。

温馨提示：要看高清无码套图，请使用手机打开并单击图片放大查看。

1.文档编写目的

使用Hue可以方便的通过界面制定Oozie的工作流，支持Hive、Pig、Spark、Java、Sqoop、MapReduce、Shell等等。Spark？那能不能支持Spark2的呢，接下来本文章就主要讲述如何使用Hue创建Spark1和Spark2的Oozie工作流。

  • 内容概述

1.添加Spark2到Oozie的Share-lib

2.创建Spark2的Oozie工作流

3.创建Spark1的Oozie工作流

4.总结

  • 测试环境

1.CM和CDH版本为5.11.2

2.操作系统：RedHat7.2

3.采用sudo权限的ec2-user用户操作

2.Oozie共享库添加Spark2

1.查看当前Oozie的share-lib共享库HDFS目录

ec2-user@ip-172-31-22-86jars$ oozie admin -ooziehttp://ip-172-31-22-86.ap-southeast-1.compute.internal:11000/oozie-sharelibupdate

ShareLib update status

host =http://ec2-54-169-0-36.ap-southeast-1.compute.amazonaws.com:11000/oozie

status = Successful

sharelibDirOld =hdfs://ip-172-31-21-45.ap-southeast-1.compute.internal:8020/user/oozie/share/lib/lib_20170921070424

sharelibDirNew =hdfs://ip-172-31-21-45.ap-southeast-1.compute.internal:8020/user/oozie/share/lib/lib_20170921070424

ec2-user@ip-172-31-22-86jars$

2.在Oozie的/user/oozie/share/lib/lib_20170921070424创建spark2目录

[ec2-user@ip-172-31-22-86 ~]$ sudo -u hdfs hadoop fs -mkdir /user/oozie/share/lib/lib_20170921070424/spark2

3.向spark2目录添加spark2的jars和oozie-sharelib-spark*.jar

[ec2-user@ip-172-31-22-86 jars]$ pwd
/opt/cloudera/parcels/SPARK2/lib/spark2/jars
[ec2-user@ip-172-31-22-86 jars]$ sudo -u hdfs hadoop fs -put *.jar /user/oozie/share/lib/lib_20170921070424/spark2
[ec2-user@ip-172-31-22-86 spark]$ pwd
/opt/cloudera/parcels/CDH/lib/oozie/oozie-sharelib-yarn/lib/spark
[ec2-user@ip-172-31-22-86 spark]$ sudo -u hdfs hadoop fs -put oozie-sharelib-spark*.jar /user/oozie/share/lib/lib_20170921070424/spark2

4.修改目录属主及权限

[ec2-user@ip-172-31-22-86 ~]$ sudo -u hdfs hadoop fs –chown -R oozie:oozie /user/oozie/share/lib/lib_20170921070424/spark2
[ec2-user@ip-172-31-22-86 ~]$ sudo -u hdfs hadoop fs –chmod -R 775 /user/oozie/share/lib/lib_20170921070424/spark2

5.更新Oozie的share-lib

[ec2-user@ip-172-31-22-86 spark]$ oozie admin -oozie http://ip-172-31-22-86.ap-southeast-1.compute.internal:11000/oozie -sharelibupdate
[ShareLib update status]
        host = http://ec2-54-169-0-36.ap-southeast-1.compute.amazonaws.com:11000/oozie
        status = Successful
        sharelibDirOld = hdfs://ip-172-31-21-45.ap-southeast-1.compute.internal:8020/user/oozie/share/lib/lib_20170921070424
        sharelibDirNew = hdfs://ip-172-31-21-45.ap-southeast-1.compute.internal:8020/user/oozie/share/lib/lib_20170921070424

[ec2-user@ip-172-31-22-86 spark]$

6.确认spark2已经添加到共享库

[ec2-user@ip-172-31-22-86 spark]$ oozie admin -oozie http://ip-172-31-22-86.ap-southeast-1.compute.internal:11000/oozie -shareliblist
[Available ShareLib]
spark2
oozie
hive
distcp
hcatalog
sqoop
mapreduce-streaming
spark
hive2
pig
[ec2-user@ip-172-31-22-86 spark]$

3.创建Spark2的Oozie工作流

1.登录Hue，创建Oozie工作流

2.进入WorkSpace

点击lib

在命令行将Spark2自带的example例子上传到/user/hue/oozie/workspaces/hue-oozie-1507832616.28/lib目录

[ec2-user@ip-172-31-22-86 jars]$ pwd
/opt/cloudera/parcels/SPARK2/lib/spark2/examples/jars
[ec2-user@ip-172-31-22-86 jars]$ sudo -u hdfs hadoop fs -put spark-examples_2.11-2.1.0.cloudera1.jar /user/hue/oozie/workspaces/hue-oozie-1507832616.28/lib
[ec2-user@ip-172-31-22-86 jars]$

3.添加Spark2任务

设置使用Spark2，否则默认使用的Spark1

完成配置，点击保存

4.保存完成后，点击运行测试是否正常

运行成功

4.创建Spark1的Oozie工作流

1.创建Oozie工作流

进入WorkSpace

2.将Spark1的example的jar包上传至该WorkSpace的lib目录

[ec2-user@ip-172-31-22-86 lib]$ pwd
/opt/cloudera/parcels/CDH/lib/spark/examples/lib
[ec2-user@ip-172-31-22-86 lib]$ sudo -u hdfs hadoop fs -put spark-examples-1.6.0-cdh5.11.2-hadoop2.6.0-cdh5.11.2.jar /user/hue/oozie/workspaces/hue-oozie-1507860705.24/lib
[ec2-user@ip-172-31-22-86 lib]$

3.添加Spark1的任务到Oozie

4.保存Oozie，然后点击提交

运行成功

5.常见问题

1.在使用Hue创建Spark2的Oozie工作流时运行异常

2017-10-16 23:20:07,086 WARN org.apache.oozie.action.hadoop.SparkActionExecutor: SERVER[ec2-54-179-152-169.ap-southeast-1.compute.amazonaws.com] USER[admin] GROUP[-] TOKEN[] APP[MyFirstSpark2] JOB[0000000-171016230402705-oozie-oozi-W] ACTION[0000000-171016230402705-oozie-oozi-W@spark-1411] Launcher exception: Exception when registering SparkListener
org.apache.spark.SparkException: Exception when registering SparkListener
   at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2193)
   at org.apache.spark.SparkContext.<init>(SparkContext.scala:562)
   at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
   at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
   at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
   at scala.Option.getOrElse(Option.scala:121)
   at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
   at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
   at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
   at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
   at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
   at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
   at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
   at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:178)
   at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:90)
   at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:81)
   at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:57)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:235)
   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:415)
   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: com.cloudera.spark.lineage.ClouderaNavigatorListener
   at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
   at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
   at java.lang.Class.forName0(Native Method)
   at java.lang.Class.forName(Class.java:270)
   at org.apache.spark.util.Utils$.classForName(Utils.scala:229)
   at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2159)
   at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2156)
   at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
   at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
   at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2156)
   ... 34 more

解决方法：

保存配置然后重启相应服务。

6.总结

使用Oozie创建Spark的工作流，如果需要运行Spark2的作业则需要向Oozie的Share-lib库中添加Spark2的支持，并在创建Spark2作业的时候需要指定Share-lib为spark2（Oozie默认的支持Spark1），否则会运行失败报Spark2的类找不到异常。

参考文档：

http://www.ericlin.me/2017/01/how-to-load-different-version-of-spark-into-oozie/?nsukey=E8Jq3WToi%2B7J%2FdQl19VwXwp9T%2Bjfeb1FCMwH9THP%2FNHS2YjcQ7soCN%2FBjG93ooOoYW21QAzNjL0HAX6dY32LCwprgJZ%2BVE%2Fu5vIgLPZwRdlEufRneOIa24y1AHXAYXeLyYXSX%2FNZdFl1h7aqol5BNKOl%2BKX5RmAvw3JiW5UryvUb2k3bL%2FaH2PK%2BeAN74MPP

醉酒鞭名马，少年多浮夸！ 岭南浣溪沙，呕吐酒肆下！挚友不肯放，数据玩的花！ 温馨提示：要看高清无码套图，请使用手机打开并单击图片放大查看。

推荐关注Hadoop实操，第一时间，分享更多Hadoop干货，欢迎转发和分享。

原创文章，欢迎转载，转载请注明：转载自微信公众号Hadoop实操

随机文章
本站知识点必读
最近更新