How to reserve complete nodes on Sun Grid Engine?

Question

How do you use SGE to reserve complete nodes on a cluster?

I don't want 2 processors from one machine, 3 processors from another, and so on. I have a quadcore cluster and I want to reserve 4 complete machines, each having 4 slots. I cannot just specify that I want 16 slots because it does not guarantee that I will have 4 slots on 4 machines each.

Changing the allocation rule to FILL_UP isn't enough because if there are no machines that are completely idle, SGE will simply "fill up" the least loaded machines as much as possible instead of waiting for 4 idle machines and then scheduling the task.

Is there any way I can do this? Is there a better place to ask this question?

Solution 1

SGE is weird with this, and I haven't found a good way to do this in the general case. One thing that you can do, if you know the memory size of the node you want, is to qsub while reserving an amount of memory almost equal to the full capacity of the node. This will ensure it grabs a system with nothing else running on it.

Solution 2

I think I found a way, but it probably doesn't work on old SGE's like mine. It seems new version of SGE has exclusive scheduling built in.

https://web.archive.org/web/20101027190030/http://wikis.sun.com/display/gridengine62u3/Configuring+Exclusive+Scheduling

Another possibility I've considered, but quite error prone, is to use qlogin instead of qsub and manually reserve 4 slots on each desired quadcore machine. Understandably, automating this is not particularly easy or fun.

Lastly, maybe this is a situation where hostgroups can be used. So for example, creating a hostgroup with 4 quadcore machines in it and then qsubbing to this specific subset of a queue, requesting a number of processors equal to the maximum total number in the group. Unfortunately this is kind of like hardcoding and has a lot of drawbacks eg having to wait for people to vacate a particular hardcoded hostgroup and requiring changes if you want to switch to 8 instead of 4 machines etc.

Solution 3

It seems like there is this hidden command-line request to add:

-l excl=true

But you have to configure it into your SGE or OpenGridScheduler by adding it to the list of complex values (qconf -mc) and enabling each individual host (qconf -me hostname)

see this link for details: http://web.archive.org/web/20130706011021/http://docs.oracle.com/cd/E24901_01/doc.62/e21978/management.htm#autoId61

In summary:

type:

qconf -mc

and add the line:

exclusive    excl      BOOL      EXCL   YES          YES          0        1000

then:

qconf -me <host_name>

and edit then complex_values line to read:

complex_values        exclusive=true

If you have any host-specific complex_values already in there, then just comma separate them.

Solution 4

We set the allocation rule to the number of slots available on the node (in this case, 4). This means you can only start jobs with n*4 CPUs, but it will achieve the desired result: 16 CPUs will be allocated as 4 nodes with 4 CPUs each.

Solution 5

Specify the allocation rule in PE configuration as $pe_slots

This will cause all slots to be allocated on a single host

View more solutions

Comments

[Kbankole]

Thanks, I considered something similar earlier except it was with load_avg instead of memory. Basically, like you say, it should be possible to specify a hard limit on a resource, that would probably only be satisfied if the machine is idle.

[Kbankole]

After thinking about it some more, I think using resource limits is probably the best solution if your SGE doesn't support exclusive scheduling. Otherwise, the person should refer to my link to Sun's wiki. Anyways, thanks for the input!

[Rahim]

We use exclusive scheduling, it's definitely the way to go. I would upgrade GridEngine if possible.

[Alex Leach]

The pvm parallel environment does need to be configured by the administrator before it can be used btw..