Ubuntu – Trying to install Sun Grid Engine on Ubuntu 10.04 – can’t connect more execution hosts

On our cluster we use h_vmem to enforce job memory allocation.

The thing you appear to be missing is setting the available amount as a consumable resource. In qconf -mc or in the qmon complexes dialog you need to set the resource as requestable and consumable.

Then, on each host with qconf -me you need to set the amount of available memory in complex_values.

For example we have host definitions that look like:

hostname              node004
load_scaling          NONE
complex_values        h_vmem=30.9G,exclusive=1,version=3.3.0

Sure, this is totally possible. SGE queues are independent of one another, so you can assign whatever nodes you would like to each queue, letting them overlap however you wish.

To create a queue, type qconf -aq: this will open up your default editor (usually vim). Type the name of the queue as the qname, add the hosts you would like to assign in the hostlist, and for slots, add a comma-delimited list of entries of the format [hostname=numslots]. Typically the number of slots is the number of cores in the host, but you can under- or over-subscribe if you prefer. If you want the queues to overlap, just add the same hosts to multiple queues.

Note, however, that by default the overlapping queues are not aware of each others' usage. They will both cheerfully assign jobs to the same node and expect them to run.

The most common way to prevent this is to makes nodes job-exclusive, so only one job may run at a time. (This is the default in other schedulers like PBS.) SGE makes this a little complicated, and involves creating a virtual "resource" which can only be used once per node. To do this, type qconf -mc to manage consumable resources. This will open an editor listing consumable resources: add a new one called "exclusive", like so:

#name          shortcut     type         relop  requestable  consumable  default  urgency
#-----------------------------------------------------------------------------------------
exclusive      excl         BOOL         EXCL   YES          YES          1        1000

For more information, see the grid engine wiki.

You can also configure what are called subordinate queues. In this, you set one queue up so that it will automatically override the other when over a certain number of slots-per-node are assigned. To set this up, run qconf -mq queue1 and under "subordinate", specify queue2=N. Then whenever the number of slots used on a node in queue1 is over N, the job in queue2 will be suspended until the queue1 job is complete.