Debian – Virtualbox HTTP load testing, host CPU overload issues

debianjavaload-testingtomcatvirtualbox

I'm doing HTTP load testing benchmarks (using Apache Benchmark and Siege) on a small Java EE 1.7.0 / Tomcat 7.0.26 application running on a Debian Squeeze 6.0.4 x64 virtualized with Virtualbox 4.1.8. The computer host is Ubuntu 11.10 x64.

I've modified those parameters in the Tomcat server.xml :

<Connector 
    port="8080" 
    protocol="HTTP/1.1"
    connectionTimeout="200000"
    redirectPort="8443" 
    acceptCount="2000"
    maxThreads="150"
    minSpareThreads="50" />

The application executed on the server takes around 300ms.

This app is running well until a certain amount of concurrent connections like those one :

ab -n 500 -c 150 http://xx.xx.xx.xx:8080/myapp/
ab -n 1000 -c 50 http://xx.xx.xx.xx:8080/myapp/
siege -b -c 100 -r 20 http://xx.xx.xx.xx:8080/myapp/

A lot of socket connection timed out happens and this completly overload the host processor (but the CPU load inside the VM is normal).

Doing an htop on the host, i can see that the Virtualbox processus is running under 300% CPU and never come down even after the load test is finished. (I've allocated 4 processors to the VM, if I allocate only one processor, CPU load goes under 100%).

Restarting Tomcat don't do anything, i'm forced to restart the whole VM.

I've tryed to launch those ab/siege commands locally on the VM and everything goes well.

I first thought it was related to a linux network limit as explained here: Running some benchmarks using ab, and tomcat starts to really slow down
So I've modified those TCP parameters :

echo 15 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 30 > /proc/sys/net/ipv4/tcp_keepalive_intvl
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse

It seems to be better, but it continues to overload the host CPU and output socket connections time out at a certain amount of concurrent connections.

I'm wondering if this is not related to how Virtualbox handles external concurrent connections.

Best Answer

I've seen this problem before when you have a loop that causes your network traffic to be re-sent forever. Try checking to see if you're getting a absurdly high amount of traffic going through any of your network interfaces.

If that doesn't work, does the CPU usage stop when the guest OS is stopped, or do you need to shutdown the whole VM container?

Related Solutions

VirtualBox Networking – How to Set Up Networking for Host and Guest

Try this:

Setup the virtualbox to use 2 adapters:
- The first adapter is set to NAT (that will give you the internet connection).
- The second adapter is set to host only.
Start the virtual machine and assign a static IP for the second adapter in Ubuntu (for instance 192.168.56.56). The host Windows will have 192.168.56.1 as IP for the internal network (VirtualBox Host-Only Network is the name in network connections in Windows). What this will give you is being able to access the apache server on ubuntu, from windows, by going to 192.168.56.56. Also, Ubuntu will have internet access, since the first adapter (set to NAT) will take care of that.
Now, to make the connection available both ways (accessing the windows host from the ubuntu guest) there's still one more step to be performed. Windows will automatically add the virtualbox host-only network to the list of public networks and that cannot be changed. This entails that the firewall will prevent proper access.
To overcome this and not make any security breaches in your setup:
- go to the windows firewall section, in control panel,
- click on advanced settings. In the page that pops up,
- click on inbound rules (left column), then on new rule (right column). Chose custom rule, set the rule to allow all programs, and any protocol. For the scope, add in the first box (local IP addresses) 192.168.56.1, and in the second box (remote IP) 192.168.56.56. Click next, select allow the connection, next, check all profiles, next, give it a name and save.

That's it, now you have 2 way communication, with apache/any other service available as well as internet. The final step is to setup a share. Do not use the shared folders feature in virtualbox, it's quite buggy especially with windows 7 (and 64 bit). Instead use samba shares - fast and efficient.

Follow this link for how to set that up: https://wiki.ubuntu.com/MountWindowsSharesPermanently

Nginx Reverse Proxy – Mapping Hudson to a Subdomain

Here is my nginx configuration file, even though I access Hudson through https (port 443):

The trick with reverse proxy (Apache or Nginx) is to always keep the same path:
If you want to redirect to a_long_and_complex_address/hudson, keep the /hudson part: myserver/hudson.
Don't try to redirect myserver to a_long_and_complex_address/hudson: all sorts of scripts and pictures, with absolute paths ('/') will broke.
Redirect myserver/hudson to a_long_and_complex_address/hudson.

Plus, you can redirect myserver/other_services to a_long_and_complex_address/other_services as well ;)

worker_processes  1;

error_log  logs/error.log  debug;
pid        logs/nginx.pid;

events {
    worker_connections  250;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;
    #keepalive_timeout  0;
    keepalive_timeout  150;
    #gzip  on;

    port_in_redirect   on;
    proxy_redirect     off;
    proxy_set_header   Host             $host;
    proxy_set_header   X-Real-IP        $remote_addr;
    proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;

    server {
        listen       443;
        server_name  (some alias);
        # default max client body size was 1m! => Error code is 413
        # here: max 10Go
        client_max_body_size 10000m;

        ssl                  on;
        ssl_certificate      /home/xxx/.ssl/my.crt;
        ssl_certificate_key  /home/xxx/.ssl/my.key;
        ssl_session_timeout  5m;
        #ssl_protocols SSLv2 SSLv3 TLSv1; # NO: SSLv2 prohibited
        ssl_protocols  SSLv3 TLSv1;
        ssl_ciphers  ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
        ssl_prefer_server_ciphers   on;

        rewrite ^/hudson$ https://myserver/hudson/ redirect;
        location /hudson/ {
          proxy_pass https://complex_address:8xx3/hudson/;
        }
    }
}

Best Answer

Related Solutions

VirtualBox Networking – How to Set Up Networking for Host and Guest

Nginx Reverse Proxy – Mapping Hudson to a Subdomain

Related Topic