Linux – Centos Monit ulimit not working

centoslinuxmonitulimit

Centos 5.9 I have set the below in /etc/security/limits.conf but when mMonit starts my processes the ulimit is still 1024. If I do ulimit -a it correctly shows 500000 as max open files. In /etc/sysctl.conf I have fs.file-max = 512000. Also root is the use that Monit is using to start the processes. I have rebooted and have the same issue.

*         hard    nofile      500000
*         soft    nofile      500000
root      hard    nofile      500000
root      soft    nofile      500000

* - nofile 655350

If I look at the process that Monit is managing I get:

[root@server1 ~]# cat /proc/8940/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            10485760             unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             257124               257124               processes
Max open files            1024                 1024                 files
Max locked memory         32768                32768                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       257124               257124               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0

Best Answer

I needed to set ulimit -n 500000 in the init file.

Related Solutions

Linux – Getting MongoDB to use max open file descriptors in /etc/security/limits.conf

So it seems MongoDB 1.6.5 doesn't work with the limit nofile line. You need to put ulimit -n X (where X is the file descriptors you want) just after the script keyword.

    script
        ulimit -n 60000
        PORT=27027
...

I guess the method using the pre-start block is for later versions of MongoDB.

Also for note, MongoDB has a hard limit of 20,000 connections, so it doesn't matter if you set file descriptors higher than that.

Another thing learned: starting and stopping using upstart scripts is not the same as REstarting. Starting and stopping threw errors on the limit nofile command, whereas restarting did not.

Linux – where are the default ulimit values set? (linux, centos)

These "default" limits are applied by:

the Linux kernel at boot time (to the init or systemd process),
inheritance, from the parent process' limits (at fork(2) time),
PAM when the user session is opened (can replace kernel/inherited values),
systemd, especially to the processes it manages,
the process itself (can replace PAM & kernel/inherited values, see setrlimit(2)).

Normal users' processes cannot rise hard limits.

The Linux kernel

At boot time, Linux sets default limits to the init (or systemd) process, which are then inherited by all the other (children) processes. To see these limits: cat /proc/1/limits.

For example, the kernel default for maximum number of file descriptors (ulimit -n) was 1024/1024 (soft, hard), and has been raised to 1024/4096 in Linux 2.6.39.

The default maximum number of processes you're talking about is limited to approximately:

Total RAM in kB / 128

for x86 architectures (at least), but distributions sometimes change default kernel values, so check your kernel source code for kernel/fork.c, fork_init(). The "number of processes" limit is called RLIMIT_NPROC there.

PAM

Usually, to ensure user authentication at login, PAM is used along with some modules (see /etc/pam.d/login).

On Debian, the PAM module responsible for setting limits is here : /lib/security/pam_limits.so.

This library will read its configuration from /etc/security/limits.conf and /etc/security/limits.d/*.conf, but even if those files are empty, pam_limits.so might use hardcoded values that you can check within the source code.

For example, on Debian, the library has been patched so that by default, the maximum number of processes (nproc) is unlimited, and the maximum number of files (nofile) is 1024/1024:

  case RLIMIT_NOFILE:
      pl->limits[i].limit.rlim_cur = 1024;
      pl->limits[i].limit.rlim_max = 1024;

So, check your CentOS' PAM module source code (look for RLIMIT_NPROC).

However, please note that many processes will not go through PAM (usually, if they are not launched by a logged in user, like daemons and maybe cron jobs).

systemd

Nowadays, systemd is widely used, it replaces init and can also configure specific limits values, especially to the processes/daemons it manages and creates itself.

Some limits it uses by default can be manually configured in /etc/systemd/system.conf. There is more information available in the documentation.