GPU Access – How to Restrict Access to GPU

cgroupsystemdubuntu-18.04

I'm configuring a small GPU server under Ubuntu 18.04.
It should support both interactive and batch jobs.

It is dedicated to run machine learning tasks by a small team. We have also some tasks for massive parallel data processing on CPU-only.

My goal is to avoid infinite seizing of GPUs by a single individual, that could, for example, launch IPython console, issue import tensorflow in it and leave office, because his working time is over.

From the other side, it is desirable to allow users do some prototyping interactively (on CPU).

Considering all above I've installed SLURM workload manager to support job queues and scheduling.

Now I want to restrict access to GPUs from interactive sessions and move them entirely under control of SLURM.

Restricting access to /dev/nvidia[0-3], as suggested here, does not work, as SLURM changes process users from root to actual task owners.

After some googleing I've come to two variants: cgroups and udev.

Reading this topic in the systemd-devel mailing list suggests that restricting access with cgroups can be done in bypass of systemd, with just issuing commands like echo "c 195:*" > /sys/fs/cgroup/devices/ blah-blah-blah /device.deny

Nevertheless I've read in the manual that systemd supports 3 slices.

I've discovered also that processes, launched by the SLURM daemon, run in system.slice, and processes, launched by users, connected to the server using ssh or remote desktop connection, run in user.slice.

In principle, I can manually edit file /lib/systemd/system/user.slice and add DeviceAllow="char-nvidia-frontend" as described here, but this file could change after next update of packages.

So, what is the correct way to setting device.deny properties in Ubuntu 18.04?

Lennart Pottering, systemd developer says that another option to restrict access is udev.

I've started reading manuals, but it seems that udev is mostly dedicated to hard drives or USB devices, not video cards.

Can anyone share an example config setting permissions?

Best Answer

I think, I've solved my task with the environment variable CUDA_VISIBLE_DEVICES.

I've put in /etc/environment a line CUDA_VISIBLE_DEVICES=-1 and thus "kindly asked" interactive sessions to not use GPU.

Related Solutions

Systemd – How to Set Environment Variable in Service

Times change and so do best practices.

The current best way to do this is to run systemctl edit myservice, which will create an override file for you or let you edit an existing one.

In normal installations this will create a directory /etc/systemd/system/myservice.service.d, and inside that directory create a file whose name ends in .conf (typically, override.conf), and in this file you can add to or override any part of the unit shipped by the distribution.

For instance, in a file /etc/systemd/system/myservice.service.d/myenv.conf:

[Service]
Environment="SECRET=pGNqduRFkB4K9C2vijOmUDa2kPtUhArN"
Environment="ANOTHER_SECRET=JP8YLOc2bsNlrGuD6LVTq7L36obpjzxd"

Also note that if the directory exists and is empty, your service will be disabled! If you don't intend to put something in the directory, ensure that it does not exist.

For reference, the old way was:

The recommended way to do this is to create a file /etc/sysconfig/myservice which contains your variables, and then load them with EnvironmentFile.

For complete details, see Fedora's documentation on how to write a systemd script.

How to set the default memory.swappiness for all the systemd cgroups

If you want to do this for ALL cgroups, do you want to do this for the whole system? If that's the case, you can set swappiness system wide in "/etc/sysctl.conf". The line you would be editing is "vm.swappiness=" which you can set to anything from 0 to 100.

As a fair warning to anyone moving this value as a way to avoid swapping altogether, setting this very low will almost entirely disable demand paging, which is a very useful way to get idle pages out of RAM and into swap. Yes, it's "swapping", but that's not such a desperate thing as is the old traditional sense of swapping when we run out of memory. Demand paging is especially useful in large databases, and it shouldn't really slow anything down. It certainly beats swapping things out when we're out of memory last second, as demand paging helps to keep that from happening in the first place while the system isn't grinding to a halt, rather than after it is. A setting of 0 for vm.swappiness system wide can cause a system to OOM kill processes when under heavy pressure, potentially bringing the whole system down in a kernel panic.

Also worth mentioning, you must call upon the sysctl.conf file to be read after editing it. # sysctl -p reads lines that have been modified in /etc/sysctl.conf. vm.swappiness can be changed on a running system this way, or by using procfs thusly: # echo 50 > /proc/sys/vm/swappiness where 50 is any number from 0 to 100.

Best Answer

Related Solutions

Systemd – How to Set Environment Variable in Service

How to set the default memory.swappiness for all the systemd cgroups

Related Topic