Reference: Fair Scheduler

Currently, the implementation is a close match to:
http://hadoop.apache.org/docs/r1.2.1/fair_scheduler.html

NOTE: Once and if we move to Hadoop 2, Fairscheduler works a bit different there: http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html

To configure it in Qubole Service on Hadoop 0.20.1:

1. Create a XML file for the Fairscheduler configuration (see at the end of this description)
2. The Fairscheduler is a Cluster setting, so you will find it in "Edit Cluster Configuration" and for each jobs we can specify which Fairscheduler pool to use (which overrides the default Fairscheduler specified, if any)
3. There could be specific XML tag specification that could be different in Qubole (in line with older version Fairscheduler settings). Example:

We are on a slightly older/different version of the FairScheduler in FIFO
configuration use:

<fifo>true</fifo>

instead of <schedulingMode>FIFO</schedulingMode> (applicable on later versions of Hadoop Fairscheduler

4. If you restart the cluster or 'Push' the configuration (there's a
button for that in the UI) - the new FairScheduler settings will take effect.

When you run DB import/export commands - you should be able to choose a pool from the UI.

If you are running sqoop from command line (through a ShellCommand) - then you should be able to use the -Dmapred.fairscheduler.pool switch

Sample Fairscheduler XML:

<allocations>
<pool name="default">
<minMaps>72</minMaps>
<schedulingMode>FAIR</schedulingMode>
</pool>
<pool name="sqoop-conf">
<maxMaps>32</maxMaps>
<maxReduces>8</maxReduces>
<fifo>true</fifo>
</pool>
<pool name="sqoop-conf2">
<maxMaps>2</maxMaps>
<fifo>true</fifo>
</pool>
<pool name="sqoop-conf3">
<maxMaps>1</maxMaps>
<fifo>true</fifo>
</pool>
<pool name="sqoop-conf4">
<maxMaps>1</maxMaps>
<fifo>true</fifo>
</pool>
<pool name="batch">
<minMaps>216</minMaps>
<fifo>true</fifo>
</pool>
<pool name="fast-paced">
<minMaps>50</minMaps>
<schedulingMode>FAIR</schedulingMode> <minSharePreemptionTimeout>300</minSharePreemptionTimeout>
</pool>
</allocations>

Have more questions? Submit a request

Comments

Powered by Zendesk