Ssh – Jmeter distributed testing over ssh tunnel

jmeterload-testingsshssh-tunnel

While configuring jmeter distributed testing i faced a problem of remote connection to servers which are behind NAT.
Node with jmeter client has external ip, all servers are in different LANs behind NAT with no possibility of direct port forwarding.

I decided to try SSH tunnels.

On server:

> jmeter-server -Djava.rmi.server.hostname=127.0.0.1
> ssh -N -f -R  55500:127.0.0.1:55500 -R 55501:127.0.0.1:55501 -L 60000:127.0.0.1:60000 jmuser@client_node_ip -p 5000

On client:

> jmeter -n -t testcase.jmx -l testcase.jtl -Djava.rmi.server.hostname=127.0.0.1 -r

Test is successful in case of small number of concurrent VU (10-100).
But when i specify more VUs SSH tunnel fails with the following errors:

accept: Too many open files
open failed: administratively prohibited: open failed
Write Failed: broken pipe

I see many connections over port 60000.
ulimit value is quite high = 1000000 on server and client nodes.

Is there any workaround?

Best Answer

There are three things I would check:

  • monitor the number of open files the application has during testing. There are various ways to do this, but I just count the entries in the /fd sub-directory for the process in the /proc directory tree. If your application does not properly close file handles when done, this can occur.
  • Check the ulimit on open files, the value you have is extremely high for that limit. Normally, it would default to about 1024, and 65536 is extremely high.
  • Monitor the number of established connections on port 60000. These will be counted against the open file limit.

If it is the tunnel that is failing, it could be the ssh server or the ssh client that is failing. Fixing the ssh server's open file limit will likely be more difficult than fixing the ssh clients open file limit.

You may be able to use multiple tunnels to work around this limitation. It will make you jmeter script slightly more difficult as you will need to spread your connections across the tunnels.

Related Topic