I'm trying to setup an environment with ELB and 4 EC2 m1.large running Apache 2 reading from another EC2 m1.small NFS v3 server.
I'm using Apache Benchmark to measure the performance in requests per second, with the following command:
ab -k -n100 -c1 http://localhost/index.html
I am mounting the NFS share with -o noatime,_netdev
The instances are on N.California.
west-1a has 2 web servers and the NFS server.
west-1c has 2 web servers.
Whenever I run ab reading the file from the local EBS volume I get consistent results, but when I run it reading from the NFS share, the results vary more than 100% when I execute the test sequentially:
for i in $(seq 100); do ab -k -n 100 -c 1 http://localhost/index.html | grep -i "requests per second" ; done
Sample output:
Requests per second: 0.36 [#/sec] (mean)
Requests per second: 2.39 [#/sec] (mean)
Requests per second: 117.12 [#/sec] (mean)
Requests per second: 128.13 [#/sec] (mean)
Requests per second: 349.67 [#/sec] (mean)
Requests per second: 161.31 [#/sec] (mean)
Requests per second: 87.54 [#/sec] (mean)
Requests per second: 132.84 [#/sec] (mean)
Requests per second: 226.37 [#/sec] (mean)
Requests per second: 258.42 [#/sec] (mean)
Requests per second: 258.02 [#/sec] (mean)
Requests per second: 272.02 [#/sec] (mean)
Requests per second: 211.56 [#/sec] (mean)
Requests per second: 180.66 [#/sec] (mean)
Requests per second: 209.89 [#/sec] (mean)
Requests per second: 0.37 [#/sec] (mean)
Requests per second: 20.62 [#/sec] (mean)
Requests per second: 38.24 [#/sec] (mean)
Requests per second: 37.95 [#/sec] (mean)
Requests per second: 91.70 [#/sec] (mean)
Requests per second: 119.31 [#/sec] (mean)
Requests per second: 112.17 [#/sec] (mean)
Requests per second: 118.26 [#/sec] (mean)
Requests per second: 119.06 [#/sec] (mean)
Requests per second: 69.67 [#/sec] (mean)
Requests per second: 117.91 [#/sec] (mean)
Requests per second: 118.93 [#/sec] (mean)
Requests per second: 119.10 [#/sec] (mean)
Requests per second: 120.92 [#/sec] (mean)
Requests per second: 121.33 [#/sec] (mean)
Requests per second: 1.42 [#/sec] (mean)
Requests per second: 106.74 [#/sec] (mean)
Requests per second: 401.16 [#/sec] (mean)
Requests per second: 404.73 [#/sec] (mean)
Requests per second: 406.82 [#/sec] (mean)
Requests per second: 417.38 [#/sec] (mean)
Requests per second: 412.92 [#/sec] (mean)
Requests per second: 420.86 [#/sec] (mean)
Requests per second: 419.59 [#/sec] (mean)
Requests per second: 415.37 [#/sec] (mean)
Requests per second: 388.77 [#/sec] (mean)
Requests per second: 413.89 [#/sec] (mean)
Requests per second: 409.92 [#/sec] (mean)
Requests per second: 416.84 [#/sec] (mean)
Requests per second: 11.53 [#/sec] (mean)
Requests per second: 38.46 [#/sec] (mean)
Requests per second: 70.85 [#/sec] (mean)
Requests per second: 71.28 [#/sec] (mean)
Requests per second: 60.33 [#/sec] (mean)
Requests per second: 55.83 [#/sec] (mean)
Requests per second: 41.74 [#/sec] (mean)
Requests per second: 46.33 [#/sec] (mean)
Requests per second: 75.05 [#/sec] (mean)
Requests per second: 95.06 [#/sec] (mean)
Requests per second: 96.73 [#/sec] (mean)
Requests per second: 97.15 [#/sec] (mean)
Requests per second: 96.70 [#/sec] (mean)
Requests per second: 57.88 [#/sec] (mean)
My question is: could that be related to AWS network, and is there any tweak that can be done to get more consistent results?
I played around moving the NFS server to the other AZ, and that did not change the inconsistency factor.
PS: This is happening from all web servers.
Best Answer
So there is a lot of moving parts to look at regarding network storage on amazon. So going to have to break this question into a few parts. You may wanna look at iSCSI as an alternative to NFS because it is known for having its issues with things like this but I will do my best to help.
So the first thing is if your using an elb for this NFS server instance then it is already network storage. I will say again in another way your network storage has network storage. To make matters more complicated each AWS splits network (and ebs traffic since its network) based on the size of the instance so with an M1 small you get a very small amount of network performance they mark it as "LOW"
Now the next thing you also have to keep track of is the caching on each of your M1.large machines. Since its not a local disk the server is unaware if the file on the drive so it wont cache the file in memory on the local machine so each web request goes out to the m1.small and gets the data which takes time, you may want to enable caching but your mileage may very with code deploys.
The last thing I would recommend you look at is checking your Apache settings and seeing if you can make it better, "25 Apache Performance Tuning Tips", and maybe playing with MPM worker and MPM prefork
P.S At the moment your m1.small has 1.7gb of memory so just keep an eye on that its going to try to cache items in memory that are coming from disk I am not sure how large your site will be but something to keep in mind.
I am going to guess the reason you had "-c" set to 1 was because your running it on a dual core server, try running it on another machine, and try setting it to 10 or a larger number,if that does not work put it all under a ELB and then get another machine and open a lot of threads, there is a warm up period on elb just a heads up