spark on yarn提交任务时一直显示ACCEPTED

spark on yarn提交任务时一直显示ACCEPTED，过一个小时后就会出现任务失败，但在提交时shell终端显示的日志并没有报错，logs文件夹中也没有日志产生。注：spark on yarn是不需要启动spark集群的，只需要在提交任务的机器配置spark就可以了，因为任务是由hadoop来执行的，spark只负责任务的提交。

任务提交命令为

 bin/spark-submit --class org.apache.spark.examples.JavaWordCount     --master yarn-client     --num-executors 25     --executor-memory 1g     --executor-cores 1     lib/spark-examples-1.0.1-hadoop2.2.0.jar     hdfs://192.168.1.11:9000/test

spark-env.sh配置如下：

export SPARK_JAR=hdfs://192.168.1.11:9000/test1/spark-assembly-1.0.1-hadoop2.2.0.jar export SPARK_YARN_APP_JAR=hdfs://192.168.1.11:9000/test1/spark-examples-1.0.1-hadoop2.2.0.jar export HADOOP_CONF_DIR=/hadoop/etc/hadoop export SPARK_EXECUTOR_INSTANCES=1000 export SPARK_EXECUTOR_CORES=1 export SPARK_EXECUTOR_MEMORY=10000M export SPARK_DRIVER_MEMORY=20000M export SPARK_YARN_APP_NAME=spark

查看日志发现报错：

14/09/04 17:10:44 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 14/09/04 17:10:45 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/09/04 17:10:46 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/09/04 17:10:47 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

这个问题很明显没有找到resourcemanager，我们可以通过以下方法来解决：

1 在spark-env.sh中添加

export SPARK_YARN_USER_ENV="CLASSPATH=hadoop-2.3.0/etc/hadoop"（修改成自己的hadoop路径）

2 自己编写JavaWordCount程序，在程序中配置集群信息

        SparkConf sparkConf = new SparkConf().setAppName("JavaWordCount");
        sparkConf.set("mapreduce.framework.name", "yarn");
        sparkConf.set("mapreduce.jobtracker.address", "192.168.1.10:9001");
        sparkConf.set("yarn.resourcemanager.hostname", "192.168.1.10");
        sparkConf.set("yarn.resourcemanager.admin.address", "192.168.1.10:8033");
        sparkConf.set("yarn.resourcemanager.address", "192.168.1.10:8032");
        sparkConf.set("yarn.resourcemanager.resource-tracker.address", "192.168.1.10:8031");
        sparkConf.set("yarn.resourcemanager.scheduler.address", "192.168.1.10:8030");
        sparkConf.set("yarn.resourcemanager.hostname", "192.168.1.10");

然后打包，在提交spark任务时使用自己打包的jar包即可。