Linux – How to raise max no of file descriptors for daemons running on Debian Jessie

databasedebianlinuxnetworkingperformance-tuning

I'm playing with pgBouncer as a connection pooling system for PostgreSQL. My system is a 12-core with 64GB RAM and 1Gbps network interface running Debian 8.1. Now I want to raise the limit for open socket connections up to, say 10.000 concurrent clients. When doing DB benchmarks the pgbench utility blocks at about 950 concurrent clients, which seems to reach a limit of 1024 open fds like it was in good old days. I checked the fs.file-max kernel parameter and the pgbench running user's resource limit:

# sysctl fs.file-max
fs.file-max = 6598264
# su - postgres
$ ulimit -Sn
65536
$ fgrep files /proc/self/limits
Max open files            65536                65536                files
$

However, the limits from proc show that the soft-limit for max open files for pgBouncer (running as user postgres) is only 1024 max open files:

$ ps -e | fgrep pgbouncer
 9840 ?        00:00:00 pgbouncer
$ fgrep files /proc/9840/limits
Limit                     Soft Limit           Hard Limit           Units
Max open files            1024                 4096                 files
$

I tried to raise the limit by inserting ulimit -S -n 5000 in /etc/default/pgbouncer (read in by the start/stop script in /etc/init.d), but that didn't work. Then I tried nofile setting in /etc/security/limits.conf and made sure it is enabled in PAM, but to no avail.

So, where exactly does start-stop-daemon lower the nofile limit for daemon processes? I stumbled over this old bug report for Debian, but it seems that the patch never was applied.

BTW: Is fs.file-max really the replacement of former system's nofiles (note the plural) kernel variable as suggested in many blog articles regarding tuning? Makes me wonder that it is in the fs parameter section. On my IRIX system it is called rlimit_no_files_max in the resource section, which makes much more sense for me than putting it in the fs section..

What I'm doing wrong here? Where is the right place to change this parameter for daemons in Debian 8.1?

Thanks in advance,

Stefan

Best Answer

You can use "ulimit" in the "start" section of the pgbouncer init script:

/etc/init.d/pgbouncer:

    [... snip ...]
    case "$1" in
      start)
        # Check if we are still disabled in /etc/default/pgbouncer
        [ "${START:-}" = "0" ] && exit 0
        log_daemon_msg "Starting PgBouncer" $NAME
        test -d $PIDDIR || install -d -o postgres -g postgres -m 2775 $PIDDIR

        ### set whatever limits you want ###
        ulimit -n 20000

        $SSD --start --chuid $RUNASUSER --oknodo -- $OPTS 2> /dev/null
        log_end_msg $?
        ;;
    [... snip ...]

With this line added, effects should take place after a restart:

Do this, or you get yelled at:

# systemctl daemon-reload

Then:

# /etc/init.d/pgbouncer restart

UPDATE:

My original answer, which showed you can add "ulimit" to the init script, works on Debian 8. Even though Deb8 is a systemd distro, it appears the default install of pgbouncer still uses the init script.

For Centos7, which manages pgbouncer completely with systemd, you want to use a systemd.service file.

First off, there appears to be a bug in the default service file for pgbouncer on Centos7, where you need to have "Type=forked" be "Type=simple".

In the pgbouncer.service file, you can also add a "LimitNOFILE=##" in the [Service] section... thus

/etc/systemd/system/pgbouncer.service

    ## good practice to include the default service file as it may change with future updates
    .include /lib/systemd/system/pgbouncer.service

    ## change the Type= per this bug
    ## http://www.postgresql.org/message-id/554A7105.1050002@gmx.net
    Type=simple

    ## Add a service section and set the max number of open files
    [Service]
    LimitNOFILE=12345

It may be worth verifying that the max number of open files is the bottleneck. You can check your logs for, essentially, "too many open files" error messages. In each case where I exceeded the max number of allowed open files, the process complained...

/var/log/postgresql/pgbouncer.log

ERROR S: login failed: FATAL: could not open relation mapping file (...): Too many open files in system
ERROR S: login failed: FATAL: could not open file (...): Too many open files in system
ERROR S: login failed: FATAL: pipe() failed: Too many open files in system
WARNING sbuf_connect failed: Too many open files in system

/var/log/postgresql/postgresql-9.4-main.log

LOG:  out of file descriptors: Too many open files in system; release and retry
PANIC:  could not open file (...): Too many open files in system
LOG:  server process (...) was terminated by signal 6: Aborted
DETAIL:  Failed process was running: END;
LOG:  terminating any other active server processes

We don't need to worry about available FDs for pgbench because if they're aren't enough, pgbench won't run:

$ ulimit -S -n 100
$ pgbench -h 192.168.122.69 -p 6432 -U postgres -d booktown -c 200 -t 10000
You need at least 202 open files but you are only allowed to use 100.
Use limit/ulimit to increase the limit before using pgbench.
Related Topic