I am running a SmartMachine on Joyent. I beleive these are some sort of virtual machine running Solaris. We have a web application on this machine running on Apache, PHP, and MySQL. It handles our moderate amount of traffic very well. However every night since we've gone live. The site will begin to return 403 Forbidden errors until Apache is restarted. I quick look at Apache's error log reveals the following:
[Tue Oct 26 23:13:00 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Wed Oct 27 13:09:40 2010] [error] (24)Too many open files: Cannot open SSLSessionCache DBM file `/var/run/ssl_scache' for reading (fetch)
[Wed Oct 27 13:09:40 2010] [error] (24)Too many open files: Cannot open SSLSessionCache DBM file `/var/run/ssl_scache' for writing (store)
[Wed Oct 27 13:09:40 2010] [error] [client 98.25.133.36] PHP Fatal error: Unknown: Failed opening required '/home/jill/web/content/index.php' (include_path='.:/opt/local/lib/php') in Unknown on line 0, referer: https://[redacted]/presentations/present#
[Wed Oct 27 13:09:42 2010] [error] (24)Too many open files: Cannot open SSLSessionCache DBM file `/var/run/ssl_scache' for reading (fetch)
[Wed Oct 27 13:09:42 2010] [error] (24)Too many open files: Cannot open SSLSessionCache DBM file `/var/run/ssl_scache' for writing (store)
[Wed Oct 27 13:09:42 2010] [crit] [client 68.193.4.75] (24)Too many open files: /home/jill/web/content/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable, referer: https://[redacted]/presentations/present#
[Wed Oct 27 13:09:43 2010] [error] (24)Too many open files: Cannot open SSLSessionCache DBM file `/var/run/ssl_scache' for reading (fetch)
[Wed Oct 27 13:09:43 2010] [error] (24)Too many open files: Cannot open SSLSessionCache DBM file `/var/run/ssl_scache' for writing (store)
[Wed Oct 27 13:09:43 2010] [crit] [client 72.28.224.201] (24)Too many open files: /home/jill/web/content/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable, referer: https://[redacted]/presentations/present#
[Wed Oct 27 13:09:44 2010] [error] (24)Too many open files: Cannot open SSLSessionCache DBM file `/var/run/ssl_scache' for reading (fetch)
[Wed Oct 27 13:09:44 2010] [error] (24)Too many open files: Cannot open SSLSessionCache DBM file `/var/run/ssl_scache' for writing (store)
[Wed Oct 27 13:09:44 2010] [crit] [client 72.28.224.201] (24)Too many open files: /home/jill/web/content/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable, referer: https://[redacted]/presentations/present#
The last three lines then repeat for every request made to the server. I'm really at a loss to how to prevent this from happening. I tried increasing the number of files that can be open using prctl, but I must not be using it right because prctl returns 1.02K for basic when I tried to set it to 65.5K. I'm not even sure if that is a sound solution:
prctl -i process -n process.max-file-descriptor `pgrep httpd`
process: 18284: /opt/local/sbin/httpd -k start
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
process.max-file-descriptor
basic 1.02K - deny 18284
privileged 65.5K - deny -
system 2.15G max deny -
process: 18285: /opt/local/sbin/httpd -k start
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
process.max-file-descriptor
basic 1.02K - deny 18285
privileged 65.5K - deny -
system 2.15G max deny -
So what is the best way to trackdown and fix a problem like this?
UPDATE
here is the output of pfiles for the root httpd process.
[root@fe5txrad ~]# pfiles 18269
18269: /opt/local/sbin/httpd -k start
Current rlimit: 1024 file descriptors
0: S_IFCHR mode:0666 dev:304,8 ino:3020727013 uid:0 gid:3 rdev:13,2
O_RDONLY
/dev/null
1: S_IFCHR mode:0666 dev:304,8 ino:3020727013 uid:0 gid:3 rdev:13,2
O_WRONLY|O_CREAT|O_TRUNC
/dev/null
2: S_IFREG mode:0640 dev:182,65550 ino:362926 uid:0 gid:0 size:20551848
O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
/var/log/httpd/error.log
3: S_IFDOOR mode:0444 dev:313,0 ino:38 uid:0 gid:0 size:0
O_RDONLY|O_LARGEFILE FD_CLOEXEC door to nscd[18176]
/var/run/name_service_door
4: S_IFSOCK mode:0666 dev:311,0 ino:43693 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK FD_CLOEXEC
SOCK_STREAM
SO_REUSEADDR,SO_KEEPALIVE,SO_SNDBUF(49152),SO_RCVBUF(49152)
sockname: AF_INET 0.0.0.0 port: 80
5: S_IFSOCK mode:0666 dev:311,0 ino:42512 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK FD_CLOEXEC
SOCK_STREAM
SO_REUSEADDR,SO_KEEPALIVE,SO_SNDBUF(49152),SO_RCVBUF(49152)
sockname: AF_INET 0.0.0.0 port: 443
6: S_IFIFO mode:0000 dev:301,0 ino:8763127 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK FD_CLOEXEC
7: S_IFIFO mode:0000 dev:301,0 ino:8763127 uid:0 gid:0 size:0
O_RDWR FD_CLOEXEC
8: S_IFREG mode:0640 dev:182,65550 ino:362927 uid:0 gid:0 size:1450493
O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE FD_CLOEXEC
/var/log/httpd/access.log
9: S_IFREG mode:0644 dev:182,65550 ino:369102 uid:1000 gid:1000 size:528239971
O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE FD_CLOEXEC
/home/jill/logs/access_log
10: S_IFREG mode:0644 dev:182,65550 ino:369102 uid:1000 gid:1000 size:528239971
O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE FD_CLOEXEC
/home/jill/logs/access_log
11: S_IFREG mode:0644 dev:308,39 ino:3386326219 uid:0 gid:0 size:0
O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE FD_CLOEXEC
12: S_IFREG mode:0644 dev:308,39 ino:3088492558 uid:0 gid:0 size:0
O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE FD_CLOEXEC
advisory write lock set by process 7350
13: S_IFSOCK mode:0666 dev:311,0 ino:6452 uid:0 gid:0 size:0
O_RDWR FD_CLOEXEC
SOCK_STREAM
SO_SNDBUF(16384),SO_RCVBUF(5120)
sockname: AF_UNIX
Best Answer
You did not mention how many connections you have as a maximum when you start getting these errors.
You can use tools/commands like:
The first command tells you how many apache process on your system. The second command tells you how many opened files/connections apache process has. Of course, you need to replace with the real value. The third command tells you how many established connection you have.