Spark History Server issue causes HTTP Error 500

Description

When user clicks on "Spark Application UI" link from the application logs pane in the Analyze UI, it shows the following error message :

HTTP ERROR 500

Workaround

1. Check the application job ID from the logs in the Analyze UI. This
usually looks like this - "application_1468959138494_0010"

2. On S3, you should see 2 files in this path:

"<DEFLOC>/logs/hadoop/<CLUSTER_ID>/<CLUSTER_INSTANCE_ID>/spark-eventlogs/<APP_ID>"

corresponding to the application - "<APP_ID>".

For example, in the case of cluster 18587, cluster instance 213337 and
application ID application_1468959138494_0010, we see these 2 files in this
S3 location:

a.
s3://qubole-defloc/logs/hadoop/18587/213337/spark-eventlogs/application_1468959138494_0010.snappy.inprogress

b.
s3://qubole-defloc/logs/hadoop/18587/213337/spark-eventlogs/application_1468959138494_0010.snappy

3. The workaround is to remove the 1st file in 2.a above.

4. No restarts required. Access the same "Spark Application UI" from the job logs pane and it should open the Spark UI most of the times. If there is any problem, please raise a support ticket. 

Have more questions? Submit a request

Comments

Powered by Zendesk