How To: Set Job Priorities with Hadoop 1 and Hadoop 2 clusters

Description:

This article describes the steps required to run the commands with the specified priorities on Hadoop 1 and Hadoop 2 clusters.

 

How To:

In order to set priority for a job on Hadoop 1 cluster, you can use the following example:

-> SET mapred.job.priority = <priority_value>;
(Use the above statement with a query)

The different <priority_value> values supported are as follows:
VERY_LOW, LOW, NORMAL, HIGH, VERY_HIGH

 

However, in order to set priority for a job on Hadoop 2 cluster, you need to perform an additional step.

This is because, in hadoop 2 cluster, pre-emption is set to false by default.

Hence, add the following to the "Override Hadoop Configuration Variables" field on the Cluster Configuration page:

-> yarn.scheduler.fair.preemption=true

After adding this configuration, restart the cluster.

Now, you can use the following property with the command:
-> SET mapreduce.job.priority = <priority_value>;
(Use the above statement with a query)

The following <priority_value> values are supported:
VERY_LOW, LOW, NORMAL, HIGH, VERY_HIGH

Have more questions? Submit a request

Comments

Powered by Zendesk