Nginx – How to simulate high bandwidth traffic for server testing

nginxnode.jsstress-testingUbuntu

I'm working on a personal CDN-like project and I would like to stress-test my server, which is a minimal nginx+nodejs+ubuntu running on a 10Gbps VPS. I want to simulate something that looks more like intended use of the server: approximately 10GB of binary data to be served by nodejs behind nginx. Connections to the server could be http or https, every connection is persistent and asks data in 1MB chunks bursts for a total of ~10-20MB per connection. Some connections could be at 100mbps with low latency (20% of connections), some could be as slow as 100-200kbps with 500ms rtt ping times (5% of connections). My goal is to produce traffic that would be close to the 10Gbps theoretical limit and try to monitor the server stats, gather analytics to be able to see how my stuff works under the load. Clients that connect to my server are my own code as well. On all clients I want to save logs and then combine and analyze them to estimate what's general outcome of the whole system when combined requested load from my server is 50% of theoretical limit, 75%, … 90%, 100%, 120% etc. During these loads my goal is to also observe server: ram use, cpu load, connection drop rate, verify logging (to ensure that combined client records match results that the server logs).

My plan is to first test on a 1Gbps server (and 10-15 x 100mbps servers in different geographical locations to simulate clients). Since I control clients I'd just make them run certain scenario and do throttled downloads at different speeds to simulate slow clinets.

What would you do if you needed to stress test such setup? is there anything wrong in my approach and why?

Best Answer

Your approach is very good, I would just suggest adding a couple of test scenarios on top of existing harness:

  1. Start with one user and gradually increase the load until response time exceeds acceptable thresholds -so called Stress Testing
  2. Put your system under prolonged load (overnight or over the weekend) with the anticipated amount of users (or a little bit more) - so called Soak Testing - this is the best way to identify the memory leaks
  3. As an addition to the Stress Testing you might also want to see what happens when the load decreases (gets back to normal): if the system recovers itself and starts operating as expected or performance problems are fatal.

Check out 4 Endurance Testing Tips You Need to Know for more comprehensive explanation of the above techniques.

Related Topic