How To: Spark Tuning

Tuning Resources

Use the Application UI and the Spark Application UI to debug the behavior of the Cluster and Executors:

Application UI


Cluster details including utilization, memory and applications

Spark  UI


Job details including stage descriptions and task summary

Spark UI


Stage details including plan, behavioral metrics and data transfer

Spark UI


Executor details including Task breakdown and resource usage metrics

 Executor Tuning

If the Jobs are lagging and the cluster is not fully utilized consider using the value of Maximum Executors for the --max-executors parameter in the Spark Submit Command Line Options in the Analyze interface or in the Notebook Interpreter.

Cache Memory

spark.executor.memory *

Container Memory

spark.executor.memory + spark.yarn.executor.memoryOverhead

Heap Memory

Container Memory - Cache Memory

Used Memory

num-executors * Container Memory

Remaining Memory

yarn.nodemanager.resource.memory-mb - Used Memory

Maximum Executors

RoundDown(Remaining Memory / Container Memory)

 Cache Tuning

If the RDD Cache Utilization is greater than 90% consider increase the percentage to allocate more memory to the RDD Cache. Alternatively consider serializing the RDD Cache with persist() to use a collection of single byte arrays which will reduce the cache memory used.

RDD Cache Size


RDD Cache Utilization

RDD Cache Size / Cache Memory

 Task Tuning

If the Jobs are lagging and the cluster is not fully utilize consider increasing spark.executors.cores and spark.cpu.tasks to increase the number of tasks per executor. If the Tasks are experiencing out of memory failures consider increasing the Heap Memory.

Number of Tasks

spark.executor.cores / spark.task.cpus

Task Memory

Heap Memory / Number of Tasks

Have more questions? Submit a request


Powered by Zendesk