Let me get all of the ulimit -n and /etc/security/limits.conf answers out of the way by saying this: I've tried setting ulimit -n to 2048, 4096, and 65535. I have set both the soft and hard limit for nofiles to all of those values. ulimit -n shows that it has been changed. But when running the server, it can still only have 1024 open sockets. accept is failing with errno 24 (too many open connections.)
After hours of googling, asking in #debian, and reading questions on ServerFault, I have still not found a solution. Why would ulimit -n show one number, but still be restricted to 1024 open file descriptors?
Some potentially useful information: The process is being started at the terminal in a screen, in su. Yes, I have added pam_limits.so to /etc/pam.d/su. I also added it to /etc/pam.d/common-session. Still no luck.
Best Answer
Nevermind, it turns out the problem was a combination of ulimit -n and FD_SETSIZE. It seems like the application is using setrlimit to set ulimit -n to FD_SETSIZE. I fixed this by undefining and redifining __FD_SETSIZE to something larger.