Spark master的HA实战案例
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.部署zookeeper集群
博主推荐阅读: https://www.cnblogs.com/yinzhengjie2020/p/12501023.html
二.配置spark的ha模式
1>.实验环境
本篇博客的实验环境是基于Standallone模式进行部署的。 博主推荐阅读: Spark的Standalone运行模式部署实战案例: https://www.cnblogs.com/yinzhengjie2020/p/13122259.html 独立部署spark历史服务: https://www.cnblogs.com/yinzhengjie2020/p/13149871.html
2>.修改spark-env.sh文件
[root@hadoop101.yinzhengjie.org.cn ~]# egrep -v "^*#|^$" /yinzhengjie/softwares/spark/conf/spark-env.sh SPARK_MASTER_HOST=hadoop105.yinzhengjie.org.cn #注意哈，这2行配置下面会对其进行注释哟~因为我们接下来是要基于zookeeper来配置spark master的HA。 SPARK_MASTER_PORT=6000 export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=30 -Dspark.history.fs.logDirectory=hdfs://hadoop101.yinzhengjie.org.cn:9000/yinzhengjie/spark/jobhistory" [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# vim /yinzhengjie/softwares/spark/conf/spark-env.sh [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# egrep -v "^*#|^$" /yinzhengjie/softwares/spark/conf/spark-env.sh export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=30 -Dspark.history.fs.logDirectory=hdfs://hadoop101.yinzhengjie.org.cn:9000/yinzhengjie/spark/jobhistory" export SPARK_DAEMON_JAVA_OPTS=" -Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=hadoop101.yinzhengjie.org.cn,hadoop102.yinzhengjie.org.cn,hadoop103.yinzhengjie.org.cn -Dspark.deploy.zookeeper.dir=/spark" [root@hadoop101.yinzhengjie.org.cn ~]#
2>.分发修改过的配置文件
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh /yinzhengjie/softwares/spark/conf/spark-env.sh ******* [hadoop102.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/spark/conf/spark-env.sh] ******* 命令执行成功 ******* [hadoop103.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/spark/conf/spark-env.sh] ******* 命令执行成功 ******* [hadoop104.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/spark/conf/spark-env.sh] ******* 命令执行成功 ******* [hadoop105.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/spark/conf/spark-env.sh] ******* 命令执行成功 ******* [hadoop106.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/spark/conf/spark-env.sh] ******* 命令执行成功 [root@hadoop101.yinzhengjie.org.cn ~]#
3>.在hadoop105.yinzhengjie.org.cn节点上重启spark集群，观察zookeeper集群的znode状态变化
[root@hadoop105.yinzhengjie.org.cn ~]# /yinzhengjie/softwares/spark/sbin/stop-all.sh hadoop101.yinzhengjie.org.cn: stopping org.apache.spark.deploy.worker.Worker hadoop102.yinzhengjie.org.cn: stopping org.apache.spark.deploy.worker.Worker hadoop106.yinzhengjie.org.cn: stopping org.apache.spark.deploy.worker.Worker hadoop103.yinzhengjie.org.cn: stopping org.apache.spark.deploy.worker.Worker hadoop104.yinzhengjie.org.cn: stopping org.apache.spark.deploy.worker.Worker stopping org.apache.spark.deploy.master.Master [root@hadoop105.yinzhengjie.org.cn ~]#
[root@hadoop105.yinzhengjie.org.cn ~]# /yinzhengjie/softwares/spark/sbin/start-all.sh starting org.apache.spark.deploy.master.Master, logging to /yinzhengjie/softwares/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-hadoop105.yinzhengjie.org.cn.out hadoop102.yinzhengjie.org.cn: starting org.apache.spark.deploy.worker.Worker, logging to /yinzhengjie/softwares/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-hadoop102.yinzhengjie.org.cn.out hadoop103.yinzhengjie.org.cn: starting org.apache.spark.deploy.worker.Worker, logging to /yinzhengjie/softwares/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-hadoop103.yinzhengjie.org.cn.out hadoop104.yinzhengjie.org.cn: starting org.apache.spark.deploy.worker.Worker, logging to /yinzhengjie/softwares/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-hadoop104.yinzhengjie.org.cn.out hadoop101.yinzhengjie.org.cn: starting org.apache.spark.deploy.worker.Worker, logging to /yinzhengjie/softwares/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-hadoop101.yinzhengjie.org.cn.out hadoop106.yinzhengjie.org.cn: starting org.apache.spark.deploy.worker.Worker, logging to /yinzhengjie/softwares/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-hadoop106.yinzhengjie.org.cn.out [root@hadoop105.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6070 Jps 5643 DataNode 5951 Master hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6272 Jps 6178 Worker 5747 DataNode 5811 JournalNode 5673 QuorumPeerMain hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5667 QuorumPeerMain 5811 JournalNode 5748 DataNode 6217 Jps 6124 Worker hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6133 Jps 5750 JournalNode 5687 DataNode 6040 Worker hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 8081 Worker 7187 HistoryServer 6388 DataNode 8244 Jps 6108 QuorumPeerMain 6271 NameNode 6687 DFSZKFailoverController 7999 ZooKeeperMain hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6672 Jps 5684 NameNode 5751 DataNode 5865 DFSZKFailoverController 6573 Worker [root@hadoop101.yinzhengjie.org.cn ~]#
4>.在hadoop101.yinzhengjie.org.cn节点上单独启动spark master角色
[root@hadoop101.yinzhengjie.org.cn ~]# jps 8448 Jps 8081 Worker 7187 HistoryServer 6388 DataNode 6108 QuorumPeerMain 8349 ZooKeeperMain 6271 NameNode 6687 DFSZKFailoverController 7999 ZooKeeperMain [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# /yinzhengjie/softwares/spark/sbin/start-master.sh starting org.apache.spark.deploy.master.Master, logging to /yinzhengjie/softwares/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-hadoop101.yinzhengjie.org.cn.out [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# jps 8081 Worker 7187 HistoryServer 8483 Master 6388 DataNode 6108 QuorumPeerMain 8540 Jps 8349 ZooKeeperMain 6271 NameNode 6687 DFSZKFailoverController 7999 ZooKeeperMain [root@hadoop101.yinzhengjie.org.cn ~]#
5>.访问spark HA集群
[root@hadoop101.yinzhengjie.org.cn ~]# spark-submit > --class org.apache.spark.examples.SparkPi > --master spark://hadoop101.yinzhengjie.org.cn:7077,hadoop105.yinzhengjie.org.cn:7077 > --executor-memory 1G > --total-executor-cores 2 > /yinzhengjie/softwares/spark/examples/jars/spark-examples_2.11-2.4.6.jar > 100 20/07/03 05:32:17 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 20/07/03 05:32:17 INFO SparkContext: Running Spark version 2.4.6 20/07/03 05:32:17 INFO SparkContext: Submitted application: Spark Pi 20/07/03 05:32:17 INFO SecurityManager: Changing view acls to: root 20/07/03 05:32:17 INFO SecurityManager: Changing modify acls to: root 20/07/03 05:32:17 INFO SecurityManager: Changing view acls groups to: 20/07/03 05:32:17 INFO SecurityManager: Changing modify acls groups to: 20/07/03 05:32:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 20/07/03 05:32:17 INFO Utils: Successfully started service 'sparkDriver' on port 21855. 20/07/03 05:32:17 INFO SparkEnv: Registering MapOutputTracker 20/07/03 05:32:17 INFO SparkEnv: Registering BlockManagerMaster 20/07/03 05:32:17 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 20/07/03 05:32:17 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 20/07/03 05:32:17 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-176a5464-d1f5-4bd0-a908-6172cfb57755 20/07/03 05:32:17 INFO MemoryStore: MemoryStore started with capacity 366.3 MB 20/07/03 05:32:17 INFO SparkEnv: Registering OutputCommitCoordinator 20/07/03 05:32:18 INFO Utils: Successfully started service 'SparkUI' on port 4040. 20/07/03 05:32:18 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://hadoop101.yinzhengjie.org.cn:4040 20/07/03 05:32:18 INFO SparkContext: Added JAR file:/yinzhengjie/softwares/spark/examples/jars/spark-examples_2.11-2.4.6.jar at spark://hadoop101.yinzhengjie.org.cn:21855/jars/spark-examples_2.11-2.4.6.jar with timestamp 1593725538110 20/07/03 05:32:18 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://hadoop101.yinzhengjie.org.cn:7077... 20/07/03 05:32:18 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://hadoop105.yinzhengjie.org.cn:7077... 20/07/03 05:32:18 INFO TransportClientFactory: Successfully created connection to hadoop101.yinzhengjie.org.cn/172.200.4.101:7077 after 33 ms (0 ms spent in bootstraps) 20/07/03 05:32:18 INFO TransportClientFactory: Successfully created connection to hadoop105.yinzhengjie.org.cn/172.200.4.105:7077 after 54 ms (0 ms spent in bootstraps) 20/07/03 05:32:19 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20200703053219-0000 20/07/03 05:32:19 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20200703053219-0000/0 on worker-20200703052304-172.200.4.101-30797 (172.200.4.101:30797) with 1 core(s) 20/07/03 05:32:19 INFO StandaloneSchedulerBackend: Granted executor ID app-20200703053219-0000/0 on hostPort 172.200.4.101:30797 with 1 core(s), 1024.0 MB RAM 20/07/03 05:32:19 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20200703053219-0000/1 on worker-20200703052304-172.200.4.102-36559 (172.200.4.102:36559) with 1 core(s) 20/07/03 05:32:19 INFO StandaloneSchedulerBackend: Granted executor ID app-20200703053219-0000/1 on hostPort 172.200.4.102:36559 with 1 core(s), 1024.0 MB RAM 20/07/03 05:32:19 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 14631. 20/07/03 05:32:19 INFO NettyBlockTransferService: Server created on hadoop101.yinzhengjie.org.cn:14631 20/07/03 05:32:19 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20200703053219-0000/1 is now RUNNING 20/07/03 05:32:19 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 20/07/03 05:32:19 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20200703053219-0000/0 is now RUNNING 20/07/03 05:32:19 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, hadoop101.yinzhengjie.org.cn, 14631, None) 20/07/03 05:32:19 INFO BlockManagerMasterEndpoint: Registering block manager hadoop101.yinzhengjie.org.cn:14631 with 366.3 MB RAM, BlockManagerId(driver, hadoop101.yinzhengjie.org.cn, 14631, None) 20/07/03 05:32:19 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, hadoop101.yinzhengjie.org.cn, 14631, None) 20/07/03 05:32:19 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, hadoop101.yinzhengjie.org.cn, 14631, None) 20/07/03 05:32:29 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.200.4.102:40705) with ID 1 20/07/03 05:32:31 INFO EventLoggingListener: Logging events to hdfs://hadoop101.yinzhengjie.org.cn:9000/yinzhengjie/spark/jobhistory/app-20200703053219-0000 20/07/03 05:32:31 INFO BlockManagerMasterEndpoint: Registering block manager 172.200.4.102:33771 with 366.3 MB RAM, BlockManagerId(1, 172.200.4.102, 33771, None) 20/07/03 05:32:31 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 20/07/03 05:32:34 INFO SparkContext: Starting job: reduce at SparkPi.scala:38 20/07/03 05:32:34 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 100 output partitions 20/07/03 05:32:34 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38) 20/07/03 05:32:34 INFO DAGScheduler: Parents of final stage: List() 20/07/03 05:32:34 INFO DAGScheduler: Missing parents: List() 20/07/03 05:32:34 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents 20/07/03 05:32:34 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.0 KB, free 366.3 MB) 20/07/03 05:32:34 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1381.0 B, free 366.3 MB) 20/07/03 05:32:34 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on hadoop101.yinzhengjie.org.cn:14631 (size: 1381.0 B, free: 366.3 MB) 20/07/03 05:32:34 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1163 20/07/03 05:32:34 INFO DAGScheduler: Submitting 100 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 20/07/03 05:32:34 INFO TaskSchedulerImpl: Adding task set 0.0 with 100 tasks 20/07/03 05:32:35 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 172.200.4.102, executor 1, partition 0, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:35 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.200.4.101:46187) with ID 0 20/07/03 05:32:35 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 172.200.4.101, executor 0, partition 1, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:37 INFO BlockManagerMasterEndpoint: Registering block manager 172.200.4.101:19103 with 366.3 MB RAM, BlockManagerId(0, 172.200.4.101, 19103, None) 20/07/03 05:32:43 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.200.4.102:33771 (size: 1381.0 B, free: 366.3 MB) 20/07/03 05:32:43 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.200.4.101:19103 (size: 1381.0 B, free: 366.3 MB) 20/07/03 05:32:43 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, 172.200.4.102, executor 1, partition 2, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:43 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 8714 ms on 172.200.4.102 (executor 1) (1/100) 20/07/03 05:32:43 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, 172.200.4.102, executor 1, partition 3, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:43 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 84 ms on 172.200.4.102 (executor 1) (2/100) 20/07/03 05:32:43 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, 172.200.4.102, executor 1, partition 4, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:43 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 94 ms on 172.200.4.102 (executor 1) (3/100) 20/07/03 05:32:43 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, 172.200.4.102, executor 1, partition 5, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 182 ms on 172.200.4.102 (executor 1) (4/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, 172.200.4.102, executor 1, partition 6, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 55 ms on 172.200.4.102 (executor 1) (5/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, 172.200.4.102, executor 1, partition 7, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 78 ms on 172.200.4.102 (executor 1) (6/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, 172.200.4.102, executor 1, partition 8, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 96 ms on 172.200.4.102 (executor 1) (7/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, 172.200.4.102, executor 1, partition 9, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 119 ms on 172.200.4.102 (executor 1) (8/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 10.0 in stage 0.0 (TID 10, 172.200.4.102, executor 1, partition 10, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 62 ms on 172.200.4.102 (executor 1) (9/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 11.0 in stage 0.0 (TID 11, 172.200.4.102, executor 1, partition 11, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 10.0 in stage 0.0 (TID 10) in 68 ms on 172.200.4.102 (executor 1) (10/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 12.0 in stage 0.0 (TID 12, 172.200.4.102, executor 1, partition 12, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 11.0 in stage 0.0 (TID 11) in 89 ms on 172.200.4.102 (executor 1) (11/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 13.0 in stage 0.0 (TID 13, 172.200.4.102, executor 1, partition 13, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 12.0 in stage 0.0 (TID 12) in 46 ms on 172.200.4.102 (executor 1) (12/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 14.0 in stage 0.0 (TID 14, 172.200.4.102, executor 1, partition 14, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 13.0 in stage 0.0 (TID 13) in 34 ms on 172.200.4.102 (executor 1) (13/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 15.0 in stage 0.0 (TID 15, 172.200.4.102, executor 1, partition 15, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 14.0 in stage 0.0 (TID 14) in 55 ms on 172.200.4.102 (executor 1) (14/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 16.0 in stage 0.0 (TID 16, 172.200.4.102, executor 1, partition 16, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 15.0 in stage 0.0 (TID 15) in 57 ms on 172.200.4.102 (executor 1) (15/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 17.0 in stage 0.0 (TID 17, 172.200.4.102, executor 1, partition 17, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 16.0 in stage 0.0 (TID 16) in 51 ms on 172.200.4.102 (executor 1) (16/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 18.0 in stage 0.0 (TID 18, 172.200.4.102, executor 1, partition 18, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 17.0 in stage 0.0 (TID 17) in 47 ms on 172.200.4.102 (executor 1) (17/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 19.0 in stage 0.0 (TID 19, 172.200.4.102, executor 1, partition 19, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 18.0 in stage 0.0 (TID 18) in 17 ms on 172.200.4.102 (executor 1) (18/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 20.0 in stage 0.0 (TID 20, 172.200.4.102, executor 1, partition 20, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 19.0 in stage 0.0 (TID 19) in 23 ms on 172.200.4.102 (executor 1) (19/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 21.0 in stage 0.0 (TID 21, 172.200.4.102, executor 1, partition 21, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 20.0 in stage 0.0 (TID 20) in 21 ms on 172.200.4.102 (executor 1) (20/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 22.0 in stage 0.0 (TID 22, 172.200.4.102, executor 1, partition 22, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 21.0 in stage 0.0 (TID 21) in 17 ms on 172.200.4.102 (executor 1) (21/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 23.0 in stage 0.0 (TID 23, 172.200.4.102, executor 1, partition 23, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 22.0 in stage 0.0 (TID 22) in 23 ms on 172.200.4.102 (executor 1) (22/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 24.0 in stage 0.0 (TID 24, 172.200.4.102, executor 1, partition 24, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 23.0 in stage 0.0 (TID 23) in 25 ms on 172.200.4.102 (executor 1) (23/100) 20/07/03 05:32:44 INFO TaskSetManager: Starting task 25.0 in stage 0.0 (TID 25, 172.200.4.102, executor 1, partition 25, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:44 INFO TaskSetManager: Finished task 24.0 in stage 0.0 (TID 24) in 21 ms on 172.200.4.102 (executor 1) (24/100) 20/07/03 05:32:45 INFO TaskSetManager: Starting task 26.0 in stage 0.0 (TID 26, 172.200.4.102, executor 1, partition 26, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:45 INFO TaskSetManager: Finished task 25.0 in stage 0.0 (TID 25) in 102 ms on 172.200.4.102 (executor 1) (25/100) 20/07/03 05:32:45 INFO TaskSetManager: Starting task 27.0 in stage 0.0 (TID 27, 172.200.4.102, executor 1, partition 27, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:45 INFO TaskSetManager: Finished task 26.0 in stage 0.0 (TID 26) in 38 ms on 172.200.4.102 (executor 1) (26/100) 20/07/03 05:32:45 INFO TaskSetManager: Starting task 28.0 in stage 0.0 (TID 28, 172.200.4.102, executor 1, partition 28, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:45 INFO TaskSetManager: Finished task 27.0 in stage 0.0 (TID 27) in 27 ms on 172.200.4.102 (executor 1) (27/100) 20/07/03 05:32:45 INFO TaskSetManager: Starting task 29.0 in stage 0.0 (TID 29, 172.200.4.102, executor 1, partition 29, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:45 INFO TaskSetManager: Finished task 28.0 in stage 0.0 (TID 28) in 107 ms on 172.200.4.102 (executor 1) (28/100) 20/07/03 05:32:45 INFO TaskSetManager: Starting task 30.0 in stage 0.0 (TID 30, 172.200.4.102, executor 1, partition 30, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:45 INFO TaskSetManager: Finished task 29.0 in stage 0.0 (TID 29) in 134 ms on 172.200.4.102 (executor 1) (29/100) 20/07/03 05:32:45 INFO TaskSetManager: Starting task 31.0 in stage 0.0 (TID 31, 172.200.4.102, executor 1, partition 31, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:45 INFO TaskSetManager: Finished task 30.0 in stage 0.0 (TID 30) in 332 ms on 172.200.4.102 (executor 1) (30/100) 20/07/03 05:32:45 INFO TaskSetManager: Starting task 32.0 in stage 0.0 (TID 32, 172.200.4.102, executor 1, partition 32, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:45 INFO TaskSetManager: Finished task 31.0 in stage 0.0 (TID 31) in 52 ms on 172.200.4.102 (executor 1) (31/100) 20/07/03 05:32:45 INFO TaskSetManager: Starting task 33.0 in stage 0.0 (TID 33, 172.200.4.102, executor 1, partition 33, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:45 INFO TaskSetManager: Finished task 32.0 in stage 0.0 (TID 32) in 198 ms on 172.200.4.102 (executor 1) (32/100) 20/07/03 05:32:46 INFO TaskSetManager: Starting task 34.0 in stage 0.0 (TID 34, 172.200.4.102, executor 1, partition 34, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:46 INFO TaskSetManager: Finished task 33.0 in stage 0.0 (TID 33) in 473 ms on 172.200.4.102 (executor 1) (33/100) 20/07/03 05:32:46 INFO TaskSetManager: Starting task 35.0 in stage 0.0 (TID 35, 172.200.4.102, executor 1, partition 35, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:46 INFO TaskSetManager: Finished task 34.0 in stage 0.0 (TID 34) in 401 ms on 172.200.4.102 (executor 1) (34/100) 20/07/03 05:32:46 INFO TaskSetManager: Starting task 36.0 in stage 0.0 (TID 36, 172.200.4.102, executor 1, partition 36, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:46 INFO TaskSetManager: Finished task 35.0 in stage 0.0 (TID 35) in 122 ms on 172.200.4.102 (executor 1) (35/100) 20/07/03 05:32:46 INFO TaskSetManager: Starting task 37.0 in stage 0.0 (TID 37, 172.200.4.102, executor 1, partition 37, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:46 INFO TaskSetManager: Finished task 36.0 in stage 0.0 (TID 36) in 85 ms on 172.200.4.102 (executor 1) (36/100) 20/07/03 05:32:46 INFO TaskSetManager: Starting task 38.0 in stage 0.0 (TID 38, 172.200.4.102, executor 1, partition 38, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:46 INFO TaskSetManager: Finished task 37.0 in stage 0.0 (TID 37) in 185 ms on 172.200.4.102 (executor 1) (37/100) 20/07/03 05:32:47 INFO TaskSetManager: Starting task 39.0 in stage 0.0 (TID 39, 172.200.4.102, executor 1, partition 39, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:47 INFO TaskSetManager: Finished task 38.0 in stage 0.0 (TID 38) in 261 ms on 172.200.4.102 (executor 1) (38/100) 20/07/03 05:32:47 INFO TaskSetManager: Starting task 40.0 in stage 0.0 (TID 40, 172.200.4.102, executor 1, partition 40, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:47 INFO TaskSetManager: Finished task 39.0 in stage 0.0 (TID 39) in 87 ms on 172.200.4.102 (executor 1) (39/100) 20/07/03 05:32:47 INFO TaskSetManager: Starting task 41.0 in stage 0.0 (TID 41, 172.200.4.102, executor 1, partition 41, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:47 INFO TaskSetManager: Finished task 40.0 in stage 0.0 (TID 40) in 171 ms on 172.200.4.102 (executor 1) (40/100) 20/07/03 05:32:47 INFO TaskSetManager: Starting task 42.0 in stage 0.0 (TID 42, 172.200.4.102, executor 1, partition 42, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:47 INFO TaskSetManager: Finished task 41.0 in stage 0.0 (TID 41) in 111 ms on 172.200.4.102 (executor 1) (41/100) 20/07/03 05:32:47 INFO TaskSetManager: Starting task 43.0 in stage 0.0 (TID 43, 172.200.4.101, executor 0, partition 43, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:47 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 11824 ms on 172.200.4.101 (executor 0) (42/100) 20/07/03 05:32:47 INFO TaskSetManager: Starting task 44.0 in stage 0.0 (TID 44, 172.200.4.102, executor 1, partition 44, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:47 INFO TaskSetManager: Finished task 42.0 in stage 0.0 (TID 42) in 132 ms on 172.200.4.102 (executor 1) (43/100) 20/07/03 05:32:47 INFO TaskSetManager: Starting task 45.0 in stage 0.0 (TID 45, 172.200.4.101, executor 0, partition 45, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:47 INFO TaskSetManager: Finished task 43.0 in stage 0.0 (TID 43) in 76 ms on 172.200.4.101 (executor 0) (44/100) 20/07/03 05:32:47 INFO TaskSetManager: Starting task 46.0 in stage 0.0 (TID 46, 172.200.4.101, executor 0, partition 46, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:47 INFO TaskSetManager: Finished task 45.0 in stage 0.0 (TID 45) in 66 ms on 172.200.4.101 (executor 0) (45/100) 20/07/03 05:32:47 INFO TaskSetManager: Starting task 47.0 in stage 0.0 (TID 47, 172.200.4.101, executor 0, partition 47, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:47 INFO TaskSetManager: Finished task 46.0 in stage 0.0 (TID 46) in 31 ms on 172.200.4.101 (executor 0) (46/100) 20/07/03 05:32:47 INFO TaskSetManager: Starting task 48.0 in stage 0.0 (TID 48, 172.200.4.101, executor 0, partition 48, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:47 INFO TaskSetManager: Finished task 47.0 in stage 0.0 (TID 47) in 56 ms on 172.200.4.101 (executor 0) (47/100) 20/07/03 05:32:47 INFO TaskSetManager: Starting task 49.0 in stage 0.0 (TID 49, 172.200.4.101, executor 0, partition 49, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:47 INFO TaskSetManager: Finished task 48.0 in stage 0.0 (TID 48) in 48 ms on 172.200.4.101 (executor 0) (48/100) 20/07/03 05:32:47 INFO TaskSetManager: Starting task 50.0 in stage 0.0 (TID 50, 172.200.4.101, executor 0, partition 50, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:47 INFO TaskSetManager: Finished task 49.0 in stage 0.0 (TID 49) in 42 ms on 172.200.4.101 (executor 0) (49/100) 20/07/03 05:32:47 INFO TaskSetManager: Starting task 51.0 in stage 0.0 (TID 51, 172.200.4.101, executor 0, partition 51, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:47 INFO TaskSetManager: Finished task 50.0 in stage 0.0 (TID 50) in 58 ms on 172.200.4.101 (executor 0) (50/100) 20/07/03 05:32:47 INFO TaskSetManager: Starting task 52.0 in stage 0.0 (TID 52, 172.200.4.101, executor 0, partition 52, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:47 INFO TaskSetManager: Finished task 51.0 in stage 0.0 (TID 51) in 28 ms on 172.200.4.101 (executor 0) (51/100) 20/07/03 05:32:47 INFO TaskSetManager: Starting task 53.0 in stage 0.0 (TID 53, 172.200.4.102, executor 1, partition 53, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:47 INFO TaskSetManager: Finished task 44.0 in stage 0.0 (TID 44) in 383 ms on 172.200.4.102 (executor 1) (52/100) 20/07/03 05:32:47 INFO TaskSetManager: Starting task 54.0 in stage 0.0 (TID 54, 172.200.4.101, executor 0, partition 54, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:47 INFO TaskSetManager: Finished task 52.0 in stage 0.0 (TID 52) in 46 ms on 172.200.4.101 (executor 0) (53/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 55.0 in stage 0.0 (TID 55, 172.200.4.101, executor 0, partition 55, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 54.0 in stage 0.0 (TID 54) in 26 ms on 172.200.4.101 (executor 0) (54/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 56.0 in stage 0.0 (TID 56, 172.200.4.102, executor 1, partition 56, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 57.0 in stage 0.0 (TID 57, 172.200.4.101, executor 0, partition 57, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 53.0 in stage 0.0 (TID 53) in 77 ms on 172.200.4.102 (executor 1) (55/100) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 55.0 in stage 0.0 (TID 55) in 41 ms on 172.200.4.101 (executor 0) (56/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 58.0 in stage 0.0 (TID 58, 172.200.4.101, executor 0, partition 58, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 57.0 in stage 0.0 (TID 57) in 25 ms on 172.200.4.101 (executor 0) (57/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 59.0 in stage 0.0 (TID 59, 172.200.4.101, executor 0, partition 59, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 58.0 in stage 0.0 (TID 58) in 58 ms on 172.200.4.101 (executor 0) (58/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 60.0 in stage 0.0 (TID 60, 172.200.4.101, executor 0, partition 60, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 61.0 in stage 0.0 (TID 61, 172.200.4.102, executor 1, partition 61, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 59.0 in stage 0.0 (TID 59) in 18 ms on 172.200.4.101 (executor 0) (59/100) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 56.0 in stage 0.0 (TID 56) in 100 ms on 172.200.4.102 (executor 1) (60/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 62.0 in stage 0.0 (TID 62, 172.200.4.101, executor 0, partition 62, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 60.0 in stage 0.0 (TID 60) in 55 ms on 172.200.4.101 (executor 0) (61/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 63.0 in stage 0.0 (TID 63, 172.200.4.102, executor 1, partition 63, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 61.0 in stage 0.0 (TID 61) in 60 ms on 172.200.4.102 (executor 1) (62/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 64.0 in stage 0.0 (TID 64, 172.200.4.101, executor 0, partition 64, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 62.0 in stage 0.0 (TID 62) in 72 ms on 172.200.4.101 (executor 0) (63/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 65.0 in stage 0.0 (TID 65, 172.200.4.101, executor 0, partition 65, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 66.0 in stage 0.0 (TID 66, 172.200.4.102, executor 1, partition 66, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 64.0 in stage 0.0 (TID 64) in 30 ms on 172.200.4.101 (executor 0) (64/100) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 63.0 in stage 0.0 (TID 63) in 76 ms on 172.200.4.102 (executor 1) (65/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 67.0 in stage 0.0 (TID 67, 172.200.4.101, executor 0, partition 67, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 65.0 in stage 0.0 (TID 65) in 37 ms on 172.200.4.101 (executor 0) (66/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 68.0 in stage 0.0 (TID 68, 172.200.4.101, executor 0, partition 68, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 67.0 in stage 0.0 (TID 67) in 45 ms on 172.200.4.101 (executor 0) (67/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 69.0 in stage 0.0 (TID 69, 172.200.4.102, executor 1, partition 69, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 70.0 in stage 0.0 (TID 70, 172.200.4.101, executor 0, partition 70, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 66.0 in stage 0.0 (TID 66) in 105 ms on 172.200.4.102 (executor 1) (68/100) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 68.0 in stage 0.0 (TID 68) in 33 ms on 172.200.4.101 (executor 0) (69/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 71.0 in stage 0.0 (TID 71, 172.200.4.101, executor 0, partition 71, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 70.0 in stage 0.0 (TID 70) in 22 ms on 172.200.4.101 (executor 0) (70/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 72.0 in stage 0.0 (TID 72, 172.200.4.101, executor 0, partition 72, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 71.0 in stage 0.0 (TID 71) in 40 ms on 172.200.4.101 (executor 0) (71/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 73.0 in stage 0.0 (TID 73, 172.200.4.101, executor 0, partition 73, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 72.0 in stage 0.0 (TID 72) in 40 ms on 172.200.4.101 (executor 0) (72/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 74.0 in stage 0.0 (TID 74, 172.200.4.102, executor 1, partition 74, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 69.0 in stage 0.0 (TID 69) in 112 ms on 172.200.4.102 (executor 1) (73/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 75.0 in stage 0.0 (TID 75, 172.200.4.101, executor 0, partition 75, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 73.0 in stage 0.0 (TID 73) in 55 ms on 172.200.4.101 (executor 0) (74/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 76.0 in stage 0.0 (TID 76, 172.200.4.101, executor 0, partition 76, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 75.0 in stage 0.0 (TID 75) in 17 ms on 172.200.4.101 (executor 0) (75/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 77.0 in stage 0.0 (TID 77, 172.200.4.101, executor 0, partition 77, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 76.0 in stage 0.0 (TID 76) in 16 ms on 172.200.4.101 (executor 0) (76/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 78.0 in stage 0.0 (TID 78, 172.200.4.101, executor 0, partition 78, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 77.0 in stage 0.0 (TID 77) in 37 ms on 172.200.4.101 (executor 0) (77/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 79.0 in stage 0.0 (TID 79, 172.200.4.101, executor 0, partition 79, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 78.0 in stage 0.0 (TID 78) in 38 ms on 172.200.4.101 (executor 0) (78/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 80.0 in stage 0.0 (TID 80, 172.200.4.102, executor 1, partition 80, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 81.0 in stage 0.0 (TID 81, 172.200.4.101, executor 0, partition 81, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 74.0 in stage 0.0 (TID 74) in 132 ms on 172.200.4.102 (executor 1) (79/100) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 79.0 in stage 0.0 (TID 79) in 20 ms on 172.200.4.101 (executor 0) (80/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 82.0 in stage 0.0 (TID 82, 172.200.4.101, executor 0, partition 82, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 83.0 in stage 0.0 (TID 83, 172.200.4.101, executor 0, partition 83, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 81.0 in stage 0.0 (TID 81) in 53 ms on 172.200.4.101 (executor 0) (81/100) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 82.0 in stage 0.0 (TID 82) in 20 ms on 172.200.4.101 (executor 0) (82/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 84.0 in stage 0.0 (TID 84, 172.200.4.101, executor 0, partition 84, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 83.0 in stage 0.0 (TID 83) in 15 ms on 172.200.4.101 (executor 0) (83/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 85.0 in stage 0.0 (TID 85, 172.200.4.102, executor 1, partition 85, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 80.0 in stage 0.0 (TID 80) in 256 ms on 172.200.4.102 (executor 1) (84/100) 20/07/03 05:32:48 INFO TaskSetManager: Starting task 86.0 in stage 0.0 (TID 86, 172.200.4.102, executor 1, partition 86, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:48 INFO TaskSetManager: Finished task 85.0 in stage 0.0 (TID 85) in 101 ms on 172.200.4.102 (executor 1) (85/100) 20/07/03 05:32:49 INFO TaskSetManager: Starting task 87.0 in stage 0.0 (TID 87, 172.200.4.101, executor 0, partition 87, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:49 INFO TaskSetManager: Finished task 84.0 in stage 0.0 (TID 84) in 514 ms on 172.200.4.101 (executor 0) (86/100) 20/07/03 05:32:49 INFO TaskSetManager: Starting task 88.0 in stage 0.0 (TID 88, 172.200.4.101, executor 0, partition 88, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:49 INFO TaskSetManager: Finished task 87.0 in stage 0.0 (TID 87) in 13 ms on 172.200.4.101 (executor 0) (87/100) 20/07/03 05:32:49 INFO TaskSetManager: Starting task 89.0 in stage 0.0 (TID 89, 172.200.4.101, executor 0, partition 89, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:49 INFO TaskSetManager: Starting task 90.0 in stage 0.0 (TID 90, 172.200.4.102, executor 1, partition 90, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:49 INFO TaskSetManager: Finished task 88.0 in stage 0.0 (TID 88) in 24 ms on 172.200.4.101 (executor 0) (88/100) 20/07/03 05:32:49 INFO TaskSetManager: Finished task 86.0 in stage 0.0 (TID 86) in 255 ms on 172.200.4.102 (executor 1) (89/100) 20/07/03 05:32:49 INFO TaskSetManager: Starting task 91.0 in stage 0.0 (TID 91, 172.200.4.101, executor 0, partition 91, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:49 INFO TaskSetManager: Finished task 89.0 in stage 0.0 (TID 89) in 38 ms on 172.200.4.101 (executor 0) (90/100) 20/07/03 05:32:49 INFO TaskSetManager: Starting task 92.0 in stage 0.0 (TID 92, 172.200.4.101, executor 0, partition 92, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:49 INFO TaskSetManager: Finished task 91.0 in stage 0.0 (TID 91) in 28 ms on 172.200.4.101 (executor 0) (91/100) 20/07/03 05:32:49 INFO TaskSetManager: Starting task 93.0 in stage 0.0 (TID 93, 172.200.4.101, executor 0, partition 93, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:49 INFO TaskSetManager: Finished task 92.0 in stage 0.0 (TID 92) in 51 ms on 172.200.4.101 (executor 0) (92/100) 20/07/03 05:32:49 INFO TaskSetManager: Starting task 94.0 in stage 0.0 (TID 94, 172.200.4.101, executor 0, partition 94, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:49 INFO TaskSetManager: Finished task 93.0 in stage 0.0 (TID 93) in 37 ms on 172.200.4.101 (executor 0) (93/100) 20/07/03 05:32:49 INFO TaskSetManager: Starting task 95.0 in stage 0.0 (TID 95, 172.200.4.101, executor 0, partition 95, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:49 INFO TaskSetManager: Finished task 94.0 in stage 0.0 (TID 94) in 24 ms on 172.200.4.101 (executor 0) (94/100) 20/07/03 05:32:49 INFO TaskSetManager: Starting task 96.0 in stage 0.0 (TID 96, 172.200.4.101, executor 0, partition 96, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:49 INFO TaskSetManager: Finished task 95.0 in stage 0.0 (TID 95) in 47 ms on 172.200.4.101 (executor 0) (95/100) 20/07/03 05:32:49 INFO TaskSetManager: Starting task 97.0 in stage 0.0 (TID 97, 172.200.4.101, executor 0, partition 97, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:49 INFO TaskSetManager: Finished task 96.0 in stage 0.0 (TID 96) in 59 ms on 172.200.4.101 (executor 0) (96/100) 20/07/03 05:32:49 INFO TaskSetManager: Starting task 98.0 in stage 0.0 (TID 98, 172.200.4.101, executor 0, partition 98, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:49 INFO TaskSetManager: Finished task 97.0 in stage 0.0 (TID 97) in 106 ms on 172.200.4.101 (executor 0) (97/100) 20/07/03 05:32:49 INFO TaskSetManager: Starting task 99.0 in stage 0.0 (TID 99, 172.200.4.101, executor 0, partition 99, PROCESS_LOCAL, 7870 bytes) 20/07/03 05:32:49 INFO TaskSetManager: Finished task 98.0 in stage 0.0 (TID 98) in 28 ms on 172.200.4.101 (executor 0) (98/100) 20/07/03 05:32:49 INFO TaskSetManager: Finished task 99.0 in stage 0.0 (TID 99) in 48 ms on 172.200.4.101 (executor 0) (99/100) 20/07/03 05:32:49 INFO TaskSetManager: Finished task 90.0 in stage 0.0 (TID 90) in 605 ms on 172.200.4.102 (executor 1) (100/100) 20/07/03 05:32:49 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 15.618 s 20/07/03 05:32:49 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 20/07/03 05:32:49 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 15.702906 s Pi is roughly 3.1408619140861913 20/07/03 05:32:49 INFO SparkUI: Stopped Spark web UI at http://hadoop101.yinzhengjie.org.cn:4040 20/07/03 05:32:49 INFO StandaloneSchedulerBackend: Shutting down all executors 20/07/03 05:32:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down 20/07/03 05:32:49 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 20/07/03 05:32:49 INFO MemoryStore: MemoryStore cleared 20/07/03 05:32:49 INFO BlockManager: BlockManager stopped 20/07/03 05:32:49 INFO BlockManagerMaster: BlockManagerMaster stopped 20/07/03 05:32:49 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 20/07/03 05:32:50 INFO SparkContext: Successfully stopped SparkContext 20/07/03 05:32:50 INFO ShutdownHookManager: Shutdown hook called 20/07/03 05:32:50 INFO ShutdownHookManager: Deleting directory /tmp/spark-fab827b0-90d0-40cc-a939-f02cec14576f 20/07/03 05:32:50 INFO ShutdownHookManager: Deleting directory /tmp/spark-1a233738-2c28-4962-a720-c065bf7afe14 [root@hadoop101.yinzhengjie.org.cn ~]#
6>.观察历史服务器的日志记录
