Tomcat connection errors with too many users

tomcat

I am load testing an app that runs on Tomcat (6.0.33). As I increase the number of users I start to get connection problems. Here is what jMeter returns. I am pretty new to Tomcat, can anyone give any advice on where to even start looking?

org.apache.http.conn.HttpHostConnectException: Connection to https://example.domain.edu refused
    at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
    at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
    at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
    at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:284)
    at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
    at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1075)
    at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1064)
    at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:426)
    at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:255)
    at java.lang.Thread.run(Thread.java:680)
Caused by: java.net.ConnectException: Operation timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
    at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
    at java.net.Socket.connect(Socket.java:529)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:570)
    at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:549)
    at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
    ... 12 more

Best Answer

You have here established that past a threshold number of users, your service becomes unresponsive to new connections. You need to use stats and logging at the application server to determine what causes that issue. Hanging on new connections could be the result of contention in any number of resources:

  • CPU, memory, or IO (vmstat will help with these)

  • You could be hitting a ulimit for maximum number of open sockets, files or processes (this will be logged to syslog)

  • Your application could be stuck waiting for access to an external resource, like a database, which itself could suffer from any of the resource contention issues described above.

In short: load testing, at least for the purposes of optimizing service performance under load, is only useful if you combine it with careful monitoring of the service you're testing in order to quantify its results. With that monitoring, load testing only tells you the threshold past which your service is no longer useable.

Related Topic