Reference: Fair Scheduler

Fair Scheduler Pool

Administrators may assign a cluster a default Fair Scheduler Pool if none is specified during job execution. Fair Scheduler Pool groups requests into queues and shares resources fairly between these queues. Queues can be arranged in a hierarchy to divide resources and configured with weights to share the cluster in specific proportions. Each queue may apply a different scheduling policy for distributing resources between applications and jobs. The default configuration uses memory as the driver for fair sharing however first in first out is an option as well as multi-resource with dominant resource fairness. Qubole will allow small jobs to execute to keep costs low and throughput high on shared clusters. All pools are preemptable by default therefore an administrator needs to disable this feature if required.


Fair Scheduler Policy

When defining a Fair Scheduler Policy there are several components to keep in mind and declare. Upper and lower bounds may be placed on the Mappers and Reducers for the pool which allows for the appropriate distribution of resources. A first in first out (FIFO) policy may be applied to the jobs in the pool and specific pools can be designated a preemptable. If a pool can be preempted then jobs in this pool may be delayed due to jobs coming from other pools. All pools can share a preemption timeout setting to make sure that jobs keep moving.


Sample Scheduler Policy



<pool name =”default”>

<maxMaps> 30 </maxMaps>

<maxReducers> 10 </maxReducers>

<canBePreempted> true </canBePreempted>

<fifo> false </fifo>


<pool name =”batch”>

<minMaps> 216 </minMaps>

<canBePreempted> false </canBePreempted>

<fifo> true </fifo>


<pool name=”fastlane”>

<minMaps> 50 </minMaps>

<maxMaps> 50 </maxMaps>

<fifo> false </true>





The default pool is allocated a maximum of 30 Mappers and 10 Reducers, does not resolve jobs in a First In First Out Policy and may be Preempted by other pools.

The batch pool is allocated a minimum of 216 Mappers, resolves jobs in a First In First Out policy and may not be preempted by other Pools

The fast lane is allocated a constant 50 mappers and does not resolve jobs in a First In First Out policy.

The fair share Preemption timeout is set to 60 seconds for the Fair Scheduler Policy.

Have more questions? Submit a request


Powered by Zendesk