yarn is not honouring yarn.nodemanager.resource.cpu-vcores

Question

I am using Hadoop-2.4.0 and my system configs are 24 cores, 96 GB RAM.

I am using following configs

mapreduce.map.cpu.vcores=1
yarn.nodemanager.resource.cpu-vcores=10
yarn.scheduler.minimum-allocation-vcores=1
yarn.scheduler.maximum-allocation-vcores=4
yarn.app.mapreduce.am.resource.cpu-vcores=1

yarn.nodemanager.resource.memory-mb=88064
mapreduce.map.memory.mb=3072
mapreduce.map.java.opts=-Xmx2048m

Capacity Scheduler configs

queue.default.capacity=50
queue.default.maximum_capacity=100
yarn.scheduler.capacity.root.default.user-limit-factor=2

With above configs, I expect yarn won't launch more than 10 mappers per node, but It is launching 28 mappers per node. Am I doing something wrong??

Solution

YARN is running more containers than allocated cores because by default DefaultResourceCalculator is used. It considers only memory.

public int computeAvailableContainers(Resource available, Resource required) {
// Only consider memory
return available.getMemory() / required.getMemory();
  }

Use DominantResourceCalculator, It uses both cpu and memory.

Set below config in capacity-scheduler.xml

yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator

More about DominantResourceCalculator

Comments

[Stonz2]

Good answers accompany code samples with an explanation for future readers. While the person asking this question may understand your answer, explaining how you arrived at it will help countless others.

[banjara]

@Stonz2 I apologize for incomplete answer.I asked this question and there was no answer for next 4-5 hour and it had only 5 view. Meanwhile I also debugged the code and found the answer. To help others I quickly added required configs, I had to run more experiments around it.

[jonson]

I spent a few hours trying to figure out why YARN was telling me there are a negative number of vcores available on my nodes!

[Sai Kiriti Badam]

The link to DominantResourceCalculator is broken.