Linux – Sun Grid Engine set memory requirements per jobs

gridenginelinuxmemory usageUbuntu

I want to be able to set up memory requirements per job.

For instance:
to run 5 jobs, for which I know each will need 4 GB memory. I have 16 GB RAM on the Ubuntu server and 16 GB swap. I want to avoid using the swap. Can I do something like:

qsub -l mem_required_for_my_job=4G job1
qsub -l mem_required_for_my_job=4G job2
qsub -l mem_required_for_my_job=4G job3
qsub -l mem_required_for_my_job=4G job4
qsub -l mem_required_for_my_job=4G job5

? The jobs will require 4 GB at some moment, but not in the beginning.

How to tell SGE what my requirements are? How to avoid scheduling 5 x 4 GB when only 16 GB available?

I read the user guide and tried s_vmem, h_vmem, mem_free, mem_used. None of them is what I want. I do not want my jobs to get killed in the middle of the processing. I want them not to be scheduled, unless the maximum resources needed are available.

Can I do this?

Thank you all!

Best Answer

On our cluster we use h_vmem to enforce job memory allocation.

The thing you appear to be missing is setting the available amount as a consumable resource. In qconf -mc or in the qmon complexes dialog you need to set the resource as requestable and consumable.

Then, on each host with qconf -me you need to set the amount of available memory in complex_values.

For example we have host definitions that look like:

hostname              node004
load_scaling          NONE
complex_values        h_vmem=30.9G,exclusive=1,version=3.3.0

Related Solutions

Linux – Running batch jobs and getting a peak memory usage summary for each job

The VmPeak answer looks like a good one to me... You could either append vmpeak to a file every x seconds and then find the highest value. Or, every x seconds, run something like: VmPeak = curVmPeak if (curVmPeak > VmPeak).

Overlapping queues on Sun Grid Engine

Sure, this is totally possible. SGE queues are independent of one another, so you can assign whatever nodes you would like to each queue, letting them overlap however you wish.

To create a queue, type qconf -aq: this will open up your default editor (usually vim). Type the name of the queue as the qname, add the hosts you would like to assign in the hostlist, and for slots, add a comma-delimited list of entries of the format [hostname=numslots]. Typically the number of slots is the number of cores in the host, but you can under- or over-subscribe if you prefer. If you want the queues to overlap, just add the same hosts to multiple queues.

Note, however, that by default the overlapping queues are not aware of each others' usage. They will both cheerfully assign jobs to the same node and expect them to run.

The most common way to prevent this is to makes nodes job-exclusive, so only one job may run at a time. (This is the default in other schedulers like PBS.) SGE makes this a little complicated, and involves creating a virtual "resource" which can only be used once per node. To do this, type qconf -mc to manage consumable resources. This will open an editor listing consumable resources: add a new one called "exclusive", like so:

#name          shortcut     type         relop  requestable  consumable  default  urgency
#-----------------------------------------------------------------------------------------
exclusive      excl         BOOL         EXCL   YES          YES          1        1000

For more information, see the grid engine wiki.

You can also configure what are called subordinate queues. In this, you set one queue up so that it will automatically override the other when over a certain number of slots-per-node are assigned. To set this up, run qconf -mq queue1 and under "subordinate", specify queue2=N. Then whenever the number of slots used on a node in queue1 is over N, the job in queue2 will be suspended until the queue1 job is complete.

Best Answer

Related Solutions

Linux – Running batch jobs and getting a peak memory usage summary for each job

Overlapping queues on Sun Grid Engine

Related Topic