Linux ulimit – Practical Maximum Open File Descriptors for High Volume Systems

javalinuxmax-file-descriptorsulimit

We recently began load testing our application and noticed that it ran out of file descriptors after about 24 hours.

We are running RHEL 5 on a Dell 1955:

CPU: 2 x Dual Core 2.66GHz 4MB 5150 / 1333FSB
RAM: 8GB RAM
HDD: 2 x 160GB 2.5" SATA Hard Drives

I checked the file descriptor limit and it was set at 1024. Considering that our application could potentially have about 1000 incoming connections as well as a 1000 outgoing connections, this seems quite low. Not to mention any actual files that need to be opened.

My first thought was to just increase the ulimit -n parameter by a few orders of magnitude and then re-run the test but I wanted to know any potential ramifications of setting this variable too high.

Are there any best practices towards setting this other than figuring out how many file descriptors our software can theoretically open?

Best Answer

These limits came from a time where multiple "normal" users (not apps) would share the server, and we needed ways to protect them from using too many resources.

They are very low for high performance servers and we generally set them to a very high number. (24k or so) If you need higher numbers, you also need to change the sysctl file-max option (generally limited to 40k on ubuntu and 70k on rhel) .

Setting ulimit:

# ulimit -n 99999

Sysctl max files:

#sysctl -w fs.file-max=100000

Also, and very important, you may need to check if your application has a memory/file descriptor leak. Use lsof to see all it has open to see if they are valid or not. Don't try to change your system to work around applications bugs.

Related Solutions

Debian – Too many file descriptors on Debian, ulimit -n has no effect

Nevermind, it turns out the problem was a combination of ulimit -n and FD_SETSIZE. It seems like the application is using setrlimit to set ulimit -n to FD_SETSIZE. I fixed this by undefining and redifining __FD_SETSIZE to something larger.

Linux and nginx – Understanding Max File Descriptors and Best Value for worker_rlimit_nofile

worker_rlimit_nofile will set the limit for file descriptors for the worker processes as oppose to the user running nginx. If other programs running under this user will not be able to gracefully handle running out of file descriptiors then you should set this limit slightly less then what is for the user.

First, What is using your file descriptors?

Each active connection to a client
Using proxy_pass? That will open a socket to the host:port handling these requests
Using proxy_pass to a local port? Thats another open socket. (For the owner of that process)
static files being served by nginx

Why would the limit per worker be less than the OS limit?

This is controlled by the OS because the worker is not the only process running on the machine. To change it for the user running nginx see below. It would be very bad if your workers used up all of the file descriptors available to all processes, don't set your limits so that is possible.

#/etc/sysctl.conf
#This sets the value you see when running cat  /proc/sys/fs/file-max
fs.file-max = 65536"


#/etc/security/limits.conf
#this sets the defaults for all users
* soft nofile 4096
* hard nofile 4096

#This overrides the default for user `usernamehere`
usernamehere soft nofile 10240
usernamehere hard nofile 10240

After those security limit changes I believe I still had to increase the softlimit for the user using ulimit.

How do I find out what my limit is now?

ulimit -a Will display all the limits associated with the user you run it as.

Best Answer

Related Solutions

Debian – Too many file descriptors on Debian, ulimit -n has no effect

Linux and nginx – Understanding Max File Descriptors and Best Value for worker_rlimit_nofile

Related Topic