Reference: Default yarn.scheduler.maximum-allocation-vcores

As of Hadoop 2.4, YARN introduced the concept of vcores (virtual cores). A vcore is a share of host CPU that the YARN Node Manager allocates to available resources.

yarn.scheduler.maximum-allocation-vcores is the maximum allocation for every container request at the Resource Manager, in terms of virtual CPU cores. Requests higher than this won't take effect and will get capped to this value.

The default value for yarn.scheduler.maximum-allocation-vcores in Qubole is set to twice the number of CPUs. This oversubscription assumes that CPUs are not always running a thread, and hence assigning more cores enables maximum CPU utilization.

Have more questions? Submit a request

Comments

Powered by Zendesk