Reference: Notebook Basics

Interpreters

The configuration provided for the Interpreter in Qubole will serve as the configuration for the application launched in the Spark cluster. Qubole has implemented Notebooks using Zepplin and these settings are available for modification inside the Interpreter Configuration in Qubole. After configuration developers will bind Interpreters to Notebooks instructing Qubole which application to use during execution. Interpreters are only started when the query or command in a paragraph in a Notebook demands the use of the language supported by the Interpreter. The default input language for of a Spark Notebook is Scala and the default context is SparkContext (sc). The default interpreters in Qubole provide support for Spark, Shell and Markdown languages and further modifications can be made through the use of the Node Bootstrap feature. The driver in addition to the Interpreters will always run on the Master Node therefore this needs to be considered when configuring the memory settings to support the Notebook.

spark.qubole.idle.timeout

the number of minutes after which a Spark context shuts down if no job has run in that Spark context.

zepplin.spark.useHiveContext

if this is set to true then HiveContext is available in the Notebook, otherwise SQLContext is available.

zepplin.spark.concurrentSQL

if this is set to false then only one SQL query may be executed at a time by the interpreter.

zepplin.spark.sql.maxConcurrency

the number of concurrent SQL commands that can be run by the interpreter.

 

Dependent Jars

There are two methods to integrate jars with Zepplin notebooks:

/user/lib/spark/lib

copy jars to /user/lib/spark/lib on all nodes with Bootstrap

%dep

load dependent jars directly inside of a notebook paragraph  

 

Log Files

Since the Zepplin server receives the requests to trigger and start Interpreters the logs detailing these actions are in the /usr/lib/zepplin/logs directory on the Master Node. This path also contains log files detailing the activity of the Spark interpreters.

Zepplin Server Log

<current_user_name>-<host_name>.log

Spark Driver Log

<interpreter_setting_name>-<current_user_name>-<interpreter_name>-<host_name>.log

Have more questions? Submit a request

Comments

Powered by Zendesk