How To: Increase spark.driver.maxResultSize value

Description: 

This article will describe how to increase the Spark Drive Max Result Size when encountering the following error:

`org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 1413 tasks (1024.3 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)`

How To - Analyze:

Update the value in the Spark Submit Command Line Options in the Analyze interface:

conf spark.driver.maxResultSize = ?

How To - Notebook:

Update the following value in the Notebook Interpreter:

spark.driver.maxResultSize

Have more questions? Submit a request

Comments

Powered by Zendesk