Linux – Remote jmeter-server error while getting the results back after some time

javajmeterlinuxlinux-networking

I have a question, I'm using jmeter from my pc that drives a jmeter-server that is installed on a RHEL5.8

my version of java is :

[jmeter@sem-osb logs]$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 5.8 (Tikanga)
[jmeter@sem-osb logs]$ java -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

My pc runs ArchLinux. On both I use the same version of Jmeter, that is : jmeter.JMeter: Version 2.11 r1554548

So I try to run a simple test, with 3 requests to a soap webservice, and try to simulate 2 clients that run those requests forever (I want to load the server).

After some time I receive on the jmeter-server.log, the following error :

2014/04/23 14:57:39 INFO  - jmeter.util.JMeterUtils: Setting Locale to fr_FR 
2014/04/23 14:57:39 INFO  - jmeter.util.JMeterUtils: Could not find resources for 'fr_FR', using 'fr' 
2014/04/23 14:57:39 INFO  - jmeter.JMeter: Loading user properties from: /appli/jmeter/jmeter/bin/user.properties 
2014/04/23 14:57:39 INFO  - jmeter.JMeter: Loading system properties from: /appli/jmeter/jmeter/bin/system.properties 
2014/04/23 14:57:39 INFO  - jmeter.JMeter: Setting System property: java.rmi.server.hostname=172.30.96.210 
2014/04/23 14:57:39 INFO  - jmeter.JMeter: Setting System property: server_port=1099 
2014/04/23 14:57:39 INFO  - jmeter.JMeter: Copyright (c) 1998-2014 The Apache Software Foundation 
2014/04/23 14:57:39 INFO  - jmeter.JMeter: Version 2.11 r1554548 
2014/04/23 14:57:39 INFO  - jmeter.JMeter: java.version=1.7.0_51 
2014/04/23 14:57:39 INFO  - jmeter.JMeter: java.vm.name=Java HotSpot(TM) 64-Bit Server VM 
2014/04/23 14:57:39 INFO  - jmeter.JMeter: os.name=Linux 
2014/04/23 14:57:39 INFO  - jmeter.JMeter: os.arch=amd64 
2014/04/23 14:57:39 INFO  - jmeter.JMeter: os.version=2.6.18-308.11.1.el5 
2014/04/23 14:57:39 INFO  - jmeter.JMeter: file.encoding=UTF-8 
2014/04/23 14:57:39 INFO  - jmeter.JMeter: Default Locale=français (France) 
2014/04/23 14:57:39 INFO  - jmeter.JMeter: JMeter  Locale=français (France) 
2014/04/23 14:57:39 INFO  - jmeter.JMeter: JMeterHome=/appli/jmeter/jmeter 
2014/04/23 14:57:39 INFO  - jmeter.JMeter: user.dir  =/appli/jmeter/jmeter 
2014/04/23 14:57:39 INFO  - jmeter.JMeter: PWD       =/appli/jmeter/jmeter 
2014/04/23 14:57:39 INFO  - jmeter.JMeter: IP: 172.30.96.210 Name: **** FullName: **** 
2014/04/23 14:57:40 INFO  - jmeter.engine.RemoteJMeterEngineImpl: Starting backing engine on 1099 
2014/04/23 14:57:40 INFO  - jmeter.engine.RemoteJMeterEngineImpl: Local IP address=172.30.96.210 
2014/04/23 14:57:40 INFO  - jmeter.engine.RemoteJMeterEngineImpl: IP address is a site-local address; this may cause problems with remote access.
    Can be overridden by defining the system property 'java.rmi.server.hostname' - see jmeter-server script file 
2014/04/23 14:57:40 INFO  - jmeter.engine.RemoteJMeterEngineImpl: Creating RMI registry (server.rmi.create=true) 
2014/04/23 14:57:40 INFO  - jmeter.engine.RemoteJMeterEngineImpl: Bound to registry on port 1099 
2014/04/23 14:58:00 WARN  - jmeter.engine.RemoteJMeterEngineImpl: Backing engine is null, ignoring reset 
2014/04/23 14:58:00 INFO  - jmeter.samplers.SampleEvent: List of sample_variables: [] 
2014/04/23 14:58:00 INFO  - jmeter.protocol.http.sampler.HTTPSamplerBase: Cannot find .className property for htmlParser, using default 
2014/04/23 14:58:00 INFO  - jmeter.protocol.http.sampler.HTTPSamplerBase: Parser for text/html is  
2014/04/23 14:58:00 INFO  - jmeter.protocol.http.sampler.HTTPSamplerBase: Parser for application/xhtml+xml is  
2014/04/23 14:58:00 INFO  - jmeter.protocol.http.sampler.HTTPSamplerBase: Parser for application/xml is  
2014/04/23 14:58:00 INFO  - jmeter.protocol.http.sampler.HTTPSamplerBase: Parser for text/xml is  
2014/04/23 14:58:00 INFO  - jmeter.protocol.http.sampler.HTTPSamplerBase: Parser for text/vnd.wap.wml is org.apache.jmeter.protocol.http.parser.RegexpHTMLParser 
2014/04/23 14:58:00 INFO  - jmeter.samplers.BatchSampleSender: Using batching for this run. Thresholds: num=100, time=60000 
2014/04/23 14:58:00 INFO  - jmeter.samplers.DataStrippingSampleSender: Using DataStrippingSampleSender for this run 
2014/04/23 14:58:00 INFO  - jmeter.engine.RemoteJMeterEngineImpl: Creating JMeter engine on host 172.30.96.210 base '.' 
2014/04/23 14:58:00 INFO  - jmeter.engine.RemoteJMeterEngineImpl: Remote client host: 172.30.102.135 
2014/04/23 14:58:00 INFO  - jmeter.engine.StandardJMeterEngine: Listeners will be started after enabling running version 
2014/04/23 14:58:00 INFO  - jmeter.engine.StandardJMeterEngine: To revert to the earlier behaviour, define jmeterengine.startlistenerslater=false 
2014/04/23 14:58:00 INFO  - jmeter.services.FileServer: Default base='/appli/jmeter/jmeter' 
2014/04/23 14:58:00 INFO  - jmeter.services.FileServer: Set new base='.' 
2014/04/23 14:58:00 INFO  - jmeter.engine.StandardJMeterEngine: Applying properties {} 
2014/04/23 14:58:00 INFO  - jmeter.engine.RemoteJMeterEngineImpl: Running test 
2014/04/23 14:58:00 INFO  - jmeter.engine.StandardJMeterEngine: Running the test! 
2014/04/23 14:58:00 INFO  - jmeter.samplers.SampleEvent: List of sample_variables: [] 
2014/04/23 14:58:00 INFO  - jmeter.engine.util.CompoundVariable: Note: Function class names must contain the string: '.functions.' 
2014/04/23 14:58:00 INFO  - jmeter.engine.util.CompoundVariable: Note: Function class names must not contain the string: '.gui.' 
2014/04/23 14:58:00 INFO  - jmeter.engine.StandardJMeterEngine: Starting ThreadGroup: 1 : Groupe d'unités 
2014/04/23 14:58:00 INFO  - jmeter.engine.StandardJMeterEngine: Starting 2 threads for group Groupe d'unités. 
2014/04/23 14:58:00 INFO  - jmeter.engine.StandardJMeterEngine: Test will stop on error 
2014/04/23 14:58:00 INFO  - jmeter.threads.ThreadGroup: Starting thread group number 1 threads 2 ramp-up 60 perThread 30000.0 delayedStart=false 
2014/04/23 14:58:00 INFO  - jmeter.threads.JMeterThread: jmeterthread.startearlier=true (see jmeter.properties) 
2014/04/23 14:58:00 INFO  - jmeter.threads.JMeterThread: Running PostProcessors in forward order 
2014/04/23 14:58:00 INFO  - jmeter.protocol.http.sampler.HTTPHCAbstractImpl: Local host = **** 
2014/04/23 14:58:00 INFO  - jmeter.protocol.http.sampler.HTTPHC3Impl: HTTP request retry count = 0 
2014/04/23 14:58:00 INFO  - jmeter.threads.ThreadGroup: Started thread group number 1 
2014/04/23 14:58:00 INFO  - jmeter.engine.StandardJMeterEngine: All thread groups have been started 
2014/04/23 14:58:00 INFO  - jmeter.threads.JMeterThread: Thread started: Groupe d'unités 1-1 
2014/04/23 14:58:00 INFO  - jmeter.samplers.SampleResult: Note: Sample TimeStamps are START times 
2014/04/23 14:58:00 INFO  - jmeter.samplers.SampleResult: sampleresult.default.encoding is set to ISO-8859-1 
2014/04/23 14:58:00 INFO  - jmeter.samplers.SampleResult: sampleresult.useNanoTime=true 
2014/04/23 14:58:00 INFO  - jmeter.samplers.SampleResult: sampleresult.nanoThreadSleep=5000 
2014/04/23 14:58:30 INFO  - jmeter.threads.JMeterThread: Thread started: Groupe d'unités 1-2 
2014/04/23 14:59:21 INFO  - jmeter.threads.JMeterThread: Stop Test detected by thread: Groupe d'unités 1-2 
2014/04/23 14:59:21 INFO  - jmeter.threads.JMeterThread: Thread finished: Groupe d'unités 1-2 
2014/04/23 14:59:21 INFO  - jmeter.threads.JMeterThread: Stopping: Groupe d'unités 1-2 
2014/04/23 14:59:21 INFO  - jmeter.threads.JMeterThread: Stopping: Groupe d'unités 1-1 
2014/04/23 14:59:21 ERROR - jmeter.threads.RemoteThreadsListenerWrapper:  java.rmi.ConnectIOException: Exception creating connection to: 172.30.102.135; nested exception is: 
    java.net.NoRouteToHostException: Ne peut attribuer l'adresse demandée
    at sun.rmi.transport.tcp.TCPEndpoint.newSocket(Unknown Source)
    at sun.rmi.transport.tcp.TCPChannel.createConnection(Unknown Source)
    at sun.rmi.transport.tcp.TCPChannel.newConnection(Unknown Source)
    at sun.rmi.server.UnicastRef.invoke(Unknown Source)
    at org.apache.jmeter.threads.RemoteThreadsListenerImpl_Stub.threadFinished(Unknown Source)
    at org.apache.jmeter.threads.RemoteThreadsListenerWrapper.threadFinished(RemoteThreadsListenerWrapper.java:61)
    at org.apache.jmeter.threads.JMeterThread$ThreadListenerTraverser.addNode(JMeterThread.java:599)
    at org.apache.jorphan.collections.HashTree.traverseInto(HashTree.java:961)
    at org.apache.jorphan.collections.HashTree.traverse(HashTree.java:946)
    at org.apache.jmeter.threads.JMeterThread.threadFinished(JMeterThread.java:571)
    at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:310)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.net.NoRouteToHostException: Ne peut attribuer l'adresse demandée
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
    at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
    at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
    at java.net.SocksSocketImpl.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at java.net.Socket.<init>(Unknown Source)
    at java.net.Socket.<init>(Unknown Source)
    at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(Unknown Source)
    at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(Unknown Source)
    ... 12 more

2014/04/23 14:59:21 INFO  - jmeter.threads.JMeterThread: Thread finished: Groupe d'unités 1-1 
2014/04/23 14:59:21 INFO  - jmeter.engine.StandardJMeterEngine: Notifying test listeners of end of test 
2014/04/23 14:59:21 INFO  - jmeter.samplers.DataStrippingSampleSender: Test Ended on 172.30.96.210 
2014/04/23 14:59:21 INFO  - jmeter.samplers.BatchSampleSender: Test Ended on 172.30.96.210 
2014/04/23 14:59:21 INFO  - jmeter.engine.StandardJMeterEngine: Test has ended on host 172.30.96.210 

The ip mentionned in the error, 172.30.102.135 is the ip from where i start the tests, meaning my pc. So it seems after some time the jmeter-server can't reach me anymore.
The thing is that I receive a lot of results before that, so there's nothing wrong with the network configuration.
The test is stopping because I asked for it to stop after any error, but if I don't check this option it goes on, adding more errors to the log file.
I was thinking about maybe something to change in the /etc/security/limits.conf maybe, but I'm not reaylly sure.
I'd really appreciate any idea.

If you need more details just ask.

Best Answer

I've got help from this website : http://howtojboss.com/2012/07/31/concurrent-high-throughput-performance-testing-with-jmeter/

The solution was to use this parameter, it seems that when you open too many connections the ports that are used for old ones are staying too long on TIME_WAIT state, so this helps :

echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle