Ubuntu – “Too many openfiles” error with Glassfish on Ubuntu

glassfishUbuntu

I am having issues with tuning Glassfish 2.1.1 on Ubuntu 10.04 LTS 64-bit (Amazon EC2). As soon as I change the HTTP Service thread count from the default 5 value to 100, I get the following errors in the server.log.

java.net.SocketException: Too many open files

Other errors include:

[#|2011-05-19T15:41:38.034-0500|SEVERE|sun-appserver2.1|javax.enterprise.system.tools.deployment|_ThreadID=16;_ThreadName=Timer-20;_RequestID=1bd7cd3e-0011-4ebc-95e5-487b96c76b20;|"DPL8011: autodeployment failure while deploying the application : null"|#]

[#|2011-05-19T15:41:39.555-0500|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=20;_ThreadName=Timer-1;_RequestID=0d9630b5-2752-4ffb-ac7c-1cf51920155a;|
java.lang.NullPointerException
    at com.sun.jbi.management.system.AutoAdminTask.pollAutoDirectory(AutoAdminTask.java:1031)
    at com.sun.jbi.management.system.AutoAdminTask.performAutoInstall(AutoAdminTask.java:329)
    at com.sun.jbi.management.system.AutoAdminTask.performAutoFunctions(AutoAdminTask.java:288)
    at com.sun.jbi.management.system.AdminService.heartBeat(AdminService.java:967)
    at com.sun.jbi.management.system.AdminService.handleNotification(AdminService.java:198)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1732)
    at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:257)
    at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:322)
    at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:307)
    at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:229)
    at javax.management.timer.Timer.sendNotification(Timer.java:1237)
    at javax.management.timer.Timer.notifyAlarmClock(Timer.java:1206)
    at javax.management.timer.TimerAlarmClock.run(Timer.java:1289)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)
|#]

and

[#|2011-05-19T16:30:40.228-0500|SEVERE|sun-appserver2.1|org.apache.jasper.servlet.JspServlet|_ThreadID=16;_ThreadName=httpWorkerThread-4949-48;_RequestID=63d6908e-cc09-4fa8-aac0-241e7582c42f;|PWC6117: File "/opt/glassfish-v2.1.1-b31g/lib/install/applications/admingui/adminGUI_war/header.jsp" not found|#]

and

[#|2011-05-19T16:30:40.229-0500|SEVERE|sun-appserver2.1|org.apache.jasper.servlet.JspServlet|_ThreadID=17;_ThreadName=httpWorkerThread-4949-46;_RequestID=869579eb-887d-4dc4-b0fc-edc4e41755a7;|PWC6117: File "/opt/glassfish-v2.1.1-b31g/lib/install/applications/admingui/adminGUI_war/homePage.jsp" not found|#]

Googling I found the follow resources:

http://felipeferreira.net/?p=873
http://www.netadmintools.com/art295.html

My /etc/security/limits.confg has the following configuration. And I have change the tcp settings as noted in http://mariosgaee.blogspot.com/2011/04/glassfish-211-on-linux-performance.html

*       soft    nofile          65535
*       hard    nofile          65535
*       soft    stack           unlimited
*       hard    stack           unlimited

/proc/sys/fs/file-max has a value of 762655 (I did not change this) but I did add to /etc/sysctl.conf as 'fs.file-max = 762655'

ulimit output

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Any ideas what may be causing this issue? Thanks in advance!

Best Answer

To see actual limits of running process you may use /proc , just get pid of your glassfish/java process and look at cat /proc/$PID_OF_PROCESS/limits It should be "Max open files" there. Also you can monitor number of open files with "lsof -p".

Related Topic