[杂谈]Executor-1

为了让Task对象能够重用,在Executor中,每一个分区数据都会有一个Task去进行计算,计算完以后,就要释放taslk的内存,包括堆内和堆外内存,并且,要把Thread-Local的内存也释放掉(TLA)。

org.apache.spark.scheduler.Task
run() {
try {
Utils.tryLogNonFatalError {
// Release memory used by this thread for unrolling blocks
SparkEnv.get.blockManager.memoryStore.releaseUnrollMemoryForThisTask(MemoryMode.ON_HEAP)
SparkEnv.get.blockManager.memoryStore.releaseUnrollMemoryForThisTask(MemoryMode.OFF_HEAP)
// Notify any tasks waiting for execution memory to be freed to wake up and try to
// acquire memory again. This makes impossible the scenario where a task sleeps forever
// because there are no other tasks left to notify it. Since this is safe to do but may
// not be strictly necessary, we should revisit whether we can remove this in the future.
val memoryManager = SparkEnv.get.memoryManager
memoryManager.synchronized { memoryManager.notifyAll() }
}
} finally {
TaskContext.unset()
}
}

原文地址:https://www.cnblogs.com/ivanny/p/spark_executor_task_1.html