apache-2.2lamplinuxPHP
I'm doing some load test over a LAMP server, but for very high load (high number of connections) tests the it stats to refuse the connections. What I must do for allow all the incoming connections from the same client?
This is the exception in the java Threads:
Caused by: java.net.ConnectException: Connection refused: connect
I finally found the setting that was really limiting the number of connections: net.ipv4.netfilter.ip_conntrack_max. This was set to 11,776 and whatever I set it to is the number of requests I can serve in my test before having to wait tcp_fin_timeout seconds for more connections to become available. The conntrack table is what the kernel uses to track the state of connections so once it's full, the kernel starts dropping packets and printing this in the log:
Jun 2 20:39:14 XXXX-XXX kernel: ip_conntrack: table full, dropping packet.
The next step was getting the kernel to recycle all those connections in the TIME_WAIT state rather than dropping packets. I could get that to happen either by turning on tcp_tw_recycle or increasing ip_conntrack_max to be larger than the number of local ports made available for connections by ip_local_port_range. I guess once the kernel is out of local ports it starts recycling connections. This uses more memory tracking connections but it seems like the better solution than turning on tcp_tw_recycle since the docs imply that that is dangerous.
With this configuration I can run ab all day and never run out of connections:
net.ipv4.netfilter.ip_conntrack_max = 32768
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_orphan_retries = 1
net.ipv4.tcp_fin_timeout = 25
net.ipv4.tcp_max_orphans = 8192
net.ipv4.ip_local_port_range = 32768 61000
The tcp_max_orphans setting didn't have any effect on my tests and I don't know why. I would think it would close the connections in TIME_WAIT state once there were 8192 of them but it doesn't do that for me.
(moved from the question itself to a separate answer instead)
According to a user on HN (a KVM developer?) this is due to small packet performance in Xen and also KVM. It's a known problem with virtualization and according to him, VMWare's ESX handles this much better. He also noted that KVM are bringing some new features designed alleviate this (original post).
This info is a bit discouraging if it's correct. Either way, I'll try the steps below until some Xen guru comes along with a definitive answer :)
Iain Kay from the xen-users mailing list compiled this graph:
Notice the TCP_CRR bars, compare "2.6.18-239.9.1.el5" vs "2.6.39 (with Xen 4.1.0)".
Submit this issue to a Xen-specific mailing list and the xensource's bugzilla as suggested by syneticon-dj
A message was posted to the xen-user list, awaiting reply.
Create a simple pathological, application-level test case and publish it.
A test server with instructions have been created and published to GitHub. With this you should be able to see a more real-world use-case compared to netperf.
Try a 32-bit PV Xen guest instance, as 64-bit might be causing more overhead in Xen. Someone mentioned this on HN. Did not make a difference.
Try enabling net.ipv4.tcp_syncookies in sysctl.conf as suggested by abofh on HN. This apparently might improve performance since the handshake would occur in the kernel. I had no luck with this.
Increase the backlog from 1024 to something much higher, also suggested by abofh on HN. This could also help since guest could potentially accept() more connections during it's execution slice given by dom0 (the host).
Double-check that conntrack is disabled on all machines as it can halve the accept rate (suggested by deubeulyou). Yes, it was disabled in all tests.
Check for "listen queue overflow and syncache buckets overflow in netstat -s" (suggested by mike_esspe on HN).
Split the interrupt handling among multiple cores (RPS/RFS I tried enabling earlier are supposed to do this, but could be worth trying again). Suggested by adamt at HN.
Turning off TCP segmentation offload and scatter/gather acceleration as suggested by Matt Bailey. (Not possible on EC2 or similar VPS hosts)
Best Answer
You're probably going to need to change the maxclients setting in your apache config file.
Documentation is here: http://httpd.apache.org/docs/1.3/mod/core.html#maxclients
and keep in mind that if you want to push the limit above 256 clients, you need to edit the HARD_SERVER_LIMIT