Usecases for mapred.job.queue.name

Question

What are the real world use cases on using map reduce job queues i.e. the value of mapred.job.queue.name property. I see default being used as the value always.

Solution

mapred.job.queue.name is what you use to assign a job to a particular queue. By default all jobs go to "default" queue. However, it is possible to create hierarchical queue. Like root, root.q1, root.q1.q1a and so on. Each of these queues can have its own set of attributes to ensure certain priority.

A real world scenario will be when you have multiple stakeholder asking for reports on same set of infrastructure. For example, at my work place , we have the data scientist trying to run various kind of research job, the customer support team looking for various daily and weekly figures and then there are real jobs that supports the day to day business. It is at the heart that the infrastructure should be able to support the best it can. Having various queues with different priorities just makes it easy for Hadoop to decide what to do next when a processor is available, or how much it can use.

So data scientist will assign to "Data Analyst" queue, marketing team will assign to "Marketing" queue. It is possible to change priority of a queue depending on time of the day.

Comments

[Aravind Yarram]

+1 So i See "Prioritizing" being one of the use-cases. Are there more?

[javadevg]

Other than priority, it also defines the order of execution. So if you have two jobs in a queue. Until Job1 is done or nothing is there in Job1 that can use the available processor, that Job2 kicks in.

[Aravind Yarram]

Do you have any insights on controlling security through queue? hadoop.apache.org/docs/r1.0.4/mapred-default.html

[javadevg]

Only security aspect I can see is , you can configure in such a way, that jobs from user A will always go to QueueA and jobs from user B will always go to QueueB....if you want to consider that as any level of security. Also, it is possible that User A can access only QueueA...