18/03/18 04:53:44 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your clu

scala> sc.textFile("hdfs://slaver1:9000/hello.txt").flatMap(_.split(" ")).map((_,1)).reduceByKey(_ + _).collect

[root@slaver1 spark-1.6.1-bin-hadoop2.6]# bin/spark-shell

[root@slaver1 spark-1.6.1-bin-hadoop2.6]# bin/spark-shell --master spark://slaver1:7077 --executor-memory 512M --total-executor-cores 2
18/03/18 05:03:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/03/18 05:03:18 INFO SecurityManager: Changing view acls to: root
18/03/18 05:03:18 INFO SecurityManager: Changing modify acls to: root
18/03/18 05:03:18 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
18/03/18 05:03:19 INFO HttpServer: Starting HTTP Server
18/03/18 05:03:19 INFO Utils: Successfully started service 'HTTP class server' on port 35095.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _ / _ / _ `/ __/  '_/
   /___/ .__/_,_/_/ /_/_   version 1.6.1
      /_/

Using Scala version 2.10.5 (Java HotSpot(TM) Client VM, Java 1.7.0_65)
Type in expressions to have them evaluated.
Type :help for more information.
18/03/18 05:03:24 INFO SparkContext: Running Spark version 1.6.1
18/03/18 05:03:24 WARN SparkConf: 
SPARK_WORKER_INSTANCES was detected (set to '1').
This is deprecated in Spark 1.0+.

Please instead use:
 - ./spark-submit with --num-executors to specify the number of executors
 - Or set SPARK_EXECUTOR_INSTANCES
 - spark.executor.instances to configure the number of instances in the spark config.
        
18/03/18 05:03:24 INFO SecurityManager: Changing view acls to: root
18/03/18 05:03:24 INFO SecurityManager: Changing modify acls to: root
18/03/18 05:03:24 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
18/03/18 05:03:25 INFO Utils: Successfully started service 'sparkDriver' on port 56128.
18/03/18 05:03:26 INFO Slf4jLogger: Slf4jLogger started
18/03/18 05:03:26 INFO Remoting: Starting remoting
18/03/18 05:03:26 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.19.128:35173]
18/03/18 05:03:26 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 35173.
18/03/18 05:03:26 INFO SparkEnv: Registering MapOutputTracker
18/03/18 05:03:26 INFO SparkEnv: Registering BlockManagerMaster
18/03/18 05:03:26 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-4a581245-fbf6-4741-81b1-bb295995d95a
18/03/18 05:03:26 INFO MemoryStore: MemoryStore started with capacity 517.4 MB
18/03/18 05:03:26 INFO SparkEnv: Registering OutputCommitCoordinator
18/03/18 05:03:27 INFO Utils: Successfully started service 'SparkUI' on port 4040.
18/03/18 05:03:27 INFO SparkUI: Started SparkUI at http://192.168.19.128:4040
18/03/18 05:03:27 INFO AppClient$ClientEndpoint: Connecting to master spark://slaver1:7077...
18/03/18 05:03:28 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20180318050328-0006
18/03/18 05:03:28 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40416.
18/03/18 05:03:28 INFO NettyBlockTransferService: Server created on 40416
18/03/18 05:03:28 INFO BlockManagerMaster: Trying to register BlockManager
18/03/18 05:03:28 INFO AppClient$ClientEndpoint: Executor added: app-20180318050328-0006/0 on worker-20180318032642-192.168.19.130-39494 (192.168.19.130:39494) with 1 cores
18/03/18 05:03:28 INFO SparkDeploySchedulerBackend: Granted executor ID app-20180318050328-0006/0 on hostPort 192.168.19.130:39494 with 1 cores, 512.0 MB RAM
18/03/18 05:03:28 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.19.128:40416 with 517.4 MB RAM, BlockManagerId(driver, 192.168.19.128, 40416)
18/03/18 05:03:28 INFO BlockManagerMaster: Registered BlockManager
18/03/18 05:03:28 INFO AppClient$ClientEndpoint: Executor added: app-20180318050328-0006/1 on worker-20180318032636-192.168.19.129-32847 (192.168.19.129:32847) with 1 cores
18/03/18 05:03:28 INFO SparkDeploySchedulerBackend: Granted executor ID app-20180318050328-0006/1 on hostPort 192.168.19.129:32847 with 1 cores, 512.0 MB RAM
18/03/18 05:03:28 INFO AppClient$ClientEndpoint: Executor updated: app-20180318050328-0006/0 is now RUNNING
18/03/18 05:03:28 INFO AppClient$ClientEndpoint: Executor updated: app-20180318050328-0006/1 is now RUNNING
18/03/18 05:03:29 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
18/03/18 05:03:29 INFO SparkILoop: Created spark context..
Spark context available as sc.
18/03/18 05:03:31 INFO HiveContext: Initializing execution hive, version 1.2.1
18/03/18 05:03:31 INFO ClientWrapper: Inspected Hadoop version: 2.6.0
18/03/18 05:03:31 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
18/03/18 05:03:32 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
18/03/18 05:03:32 INFO ObjectStore: ObjectStore, initialize called
18/03/18 05:03:33 INFO SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slaver3:58953) with ID 0
18/03/18 05:03:33 INFO BlockManagerMasterEndpoint: Registering block manager slaver3:44986 with 146.2 MB RAM, BlockManagerId(0, slaver3, 44986)
18/03/18 05:03:33 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
18/03/18 05:03:33 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
18/03/18 05:03:33 INFO SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slaver2:55466) with ID 1
18/03/18 05:03:33 INFO BlockManagerMasterEndpoint: Registering block manager slaver2:57065 with 146.2 MB RAM, BlockManagerId(1, slaver2, 57065)
18/03/18 05:03:33 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
18/03/18 05:03:34 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
18/03/18 05:03:36 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
18/03/18 05:03:38 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/03/18 05:03:38 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/03/18 05:03:39 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/03/18 05:03:39 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/03/18 05:03:39 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
18/03/18 05:03:39 INFO ObjectStore: Initialized ObjectStore
18/03/18 05:03:40 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
18/03/18 05:03:40 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
Java HotSpot(TM) Client VM warning: You have loaded library /tmp/libnetty-transport-native-epoll5989457308944360401.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
18/03/18 05:03:41 INFO HiveMetaStore: Added admin role in metastore
18/03/18 05:03:41 INFO HiveMetaStore: Added public role in metastore
18/03/18 05:03:41 INFO HiveMetaStore: No user is added in admin role, since config is empty
18/03/18 05:03:41 INFO HiveMetaStore: 0: get_all_databases
18/03/18 05:03:41 INFO audit: ugi=root    ip=unknown-ip-addr    cmd=get_all_databases    
18/03/18 05:03:41 INFO HiveMetaStore: 0: get_functions: db=default pat=*
18/03/18 05:03:41 INFO audit: ugi=root    ip=unknown-ip-addr    cmd=get_functions: db=default pat=*    
18/03/18 05:03:41 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
18/03/18 05:03:41 INFO SessionState: Created local directory: /tmp/c6bce663-21e6-4d05-a30a-0114773b6959_resources
18/03/18 05:03:41 INFO SessionState: Created HDFS directory: /tmp/hive/root/c6bce663-21e6-4d05-a30a-0114773b6959
18/03/18 05:03:41 INFO SessionState: Created local directory: /tmp/root/c6bce663-21e6-4d05-a30a-0114773b6959
18/03/18 05:03:41 INFO SessionState: Created HDFS directory: /tmp/hive/root/c6bce663-21e6-4d05-a30a-0114773b6959/_tmp_space.db
18/03/18 05:03:42 INFO HiveContext: default warehouse location is /user/hive/warehouse
18/03/18 05:03:42 INFO HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
18/03/18 05:03:42 INFO ClientWrapper: Inspected Hadoop version: 2.6.0
18/03/18 05:03:42 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
18/03/18 05:03:43 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
18/03/18 05:03:43 INFO ObjectStore: ObjectStore, initialize called
18/03/18 05:03:44 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
18/03/18 05:03:44 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
18/03/18 05:03:44 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
18/03/18 05:03:44 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
18/03/18 05:03:46 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
18/03/18 05:03:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/03/18 05:03:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/03/18 05:03:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/03/18 05:03:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/03/18 05:03:48 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
18/03/18 05:03:48 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
18/03/18 05:03:48 INFO ObjectStore: Initialized ObjectStore
18/03/18 05:03:48 INFO HiveMetaStore: Added admin role in metastore
18/03/18 05:03:48 INFO HiveMetaStore: Added public role in metastore
18/03/18 05:03:48 INFO HiveMetaStore: No user is added in admin role, since config is empty
18/03/18 05:03:48 INFO HiveMetaStore: 0: get_all_databases
18/03/18 05:03:48 INFO audit: ugi=root    ip=unknown-ip-addr    cmd=get_all_databases    
18/03/18 05:03:48 INFO HiveMetaStore: 0: get_functions: db=default pat=*
18/03/18 05:03:48 INFO audit: ugi=root    ip=unknown-ip-addr    cmd=get_functions: db=default pat=*    
18/03/18 05:03:48 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
18/03/18 05:03:49 INFO SessionState: Created local directory: /tmp/bcc435a7-3ad6-4cd4-9672-b11e5b6c4adc_resources
18/03/18 05:03:49 INFO SessionState: Created HDFS directory: /tmp/hive/root/bcc435a7-3ad6-4cd4-9672-b11e5b6c4adc
18/03/18 05:03:49 INFO SessionState: Created local directory: /tmp/root/bcc435a7-3ad6-4cd4-9672-b11e5b6c4adc
18/03/18 05:03:49 INFO SessionState: Created HDFS directory: /tmp/hive/root/bcc435a7-3ad6-4cd4-9672-b11e5b6c4adc/_tmp_space.db
18/03/18 05:03:49 INFO SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.

scala> 

scala> sc.textFile("hdfs://slaver1:9000/hello.txt").flatMap(_.split(" ")).map((_,1)).reduceByKey(_ + _).collect
18/03/18 05:05:16 WARN SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes
18/03/18 05:05:16 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 82.6 KB, free 82.6 KB)
18/03/18 05:05:17 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 19.3 KB, free 102.0 KB)
18/03/18 05:05:17 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.19.128:40416 (size: 19.3 KB, free: 517.4 MB)
18/03/18 05:05:17 INFO SparkContext: Created broadcast 0 from textFile at <console>:28
18/03/18 05:05:19 INFO FileInputFormat: Total input paths to process : 1
18/03/18 05:05:20 INFO SparkContext: Starting job: collect at <console>:28
18/03/18 05:05:20 INFO DAGScheduler: Registering RDD 3 (map at <console>:28)
18/03/18 05:05:20 INFO DAGScheduler: Got job 0 (collect at <console>:28) with 2 output partitions
18/03/18 05:05:20 INFO DAGScheduler: Final stage: ResultStage 1 (collect at <console>:28)
18/03/18 05:05:20 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
18/03/18 05:05:20 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
18/03/18 05:05:20 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at map at <console>:28), which has no missing parents
18/03/18 05:05:21 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.2 KB, free 106.1 KB)
18/03/18 05:05:21 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.3 KB, free 108.4 KB)
18/03/18 05:05:21 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.19.128:40416 (size: 2.3 KB, free: 517.4 MB)
18/03/18 05:05:21 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
18/03/18 05:05:21 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at map at <console>:28)
18/03/18 05:05:21 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
18/03/18 05:05:21 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, slaver3, partition 0,NODE_LOCAL, 2120 bytes)
18/03/18 05:05:22 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on slaver3:44986 (size: 2.3 KB, free: 146.2 MB)
18/03/18 05:05:22 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on slaver3:44986 (size: 19.3 KB, free: 146.2 MB)
18/03/18 05:05:24 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, slaver3, partition 1,NODE_LOCAL, 2120 bytes)
18/03/18 05:05:24 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 3056 ms on slaver3 (1/2)
18/03/18 05:05:24 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 199 ms on slaver3 (2/2)
18/03/18 05:05:24 INFO DAGScheduler: ShuffleMapStage 0 (map at <console>:28) finished in 3.242 s
18/03/18 05:05:24 INFO DAGScheduler: looking for newly runnable stages
18/03/18 05:05:24 INFO DAGScheduler: running: Set()
18/03/18 05:05:24 INFO DAGScheduler: waiting: Set(ResultStage 1)
18/03/18 05:05:24 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
18/03/18 05:05:24 INFO DAGScheduler: failed: Set()
18/03/18 05:05:24 INFO DAGScheduler: Submitting ResultStage 1 (ShuffledRDD[4] at reduceByKey at <console>:28), which has no missing parents
18/03/18 05:05:24 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.6 KB, free 111.0 KB)
18/03/18 05:05:24 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1589.0 B, free 112.6 KB)
18/03/18 05:05:24 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.19.128:40416 (size: 1589.0 B, free: 517.4 MB)
18/03/18 05:05:24 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006
18/03/18 05:05:24 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 1 (ShuffledRDD[4] at reduceByKey at <console>:28)
18/03/18 05:05:24 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks
18/03/18 05:05:24 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, slaver3, partition 0,NODE_LOCAL, 1894 bytes)
18/03/18 05:05:24 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on slaver3:44986 (size: 1589.0 B, free: 146.2 MB)
18/03/18 05:05:24 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to slaver3:58953
18/03/18 05:05:24 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 147 bytes
18/03/18 05:05:24 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3, slaver3, partition 1,NODE_LOCAL, 1894 bytes)
18/03/18 05:05:24 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 131 ms on slaver3 (1/2)
18/03/18 05:05:24 INFO DAGScheduler: ResultStage 1 (collect at <console>:28) finished in 0.153 s
18/03/18 05:05:24 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 30 ms on slaver3 (2/2)
18/03/18 05:05:24 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
18/03/18 05:05:24 INFO DAGScheduler: Job 0 finished: collect at <console>:28, took 3.883545 s
res0: Array[(String, Int)] = Array((hive,1), (hello,7), (sqoop,1), (spark,1), (hadoop,1), (storm,1), (hbase,1), (biexiansheng,1))