I have a mongo server on ec2 and its storing some collection which is having 12+ millions documents.
Their are 7 other instances trying to fetch some documents from the mongo server. Also their are some documents which are to be deleted. when I logged into the server its saying server load > 2.0 . The out put of iostat -xdk 1 20 command is pasted below
ubuntu@ip-10-28-170-37:~$ iostat -xdk 1 20
Linux 3.2.0-40-virtual (ip-10-28-170-37) 09/03/2013 _x86_64_ (2 CPU)
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.03 0.25 0.11 3.90 3.73 43.28 0.01 33.32 4.93 99.35 1.26 0.04
xvdb 0.00 1.19 0.01 0.72 0.25 7.64 21.69 0.00 1.25 2.13 1.24 0.42 0.03
xvdf 0.00 21.94 171.95 99.20 4613.53 1132.67 42.38 0.25 0.91 0.68 1.30 1.34 36.43
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 43.00 694.00 31.00 22544.00 780.00 64.34 4.65 6.40 6.34 7.74 1.37 99.60
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 35.00 699.00 26.00 22352.00 684.00 63.55 4.64 6.41 6.31 9.08 1.32 96.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 39.00 666.00 29.00 19700.00 800.00 58.99 5.48 7.88 7.81 9.38 1.44 100.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 23.00 668.00 22.00 19716.00 656.00 59.05 5.29 7.67 7.63 8.73 1.40 96.80
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 3.00 1.00 1.00 4.00 16.00 20.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 39.00 675.00 28.00 21040.00 788.00 62.10 5.32 7.56 7.44 10.43 1.39 98.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 4.00 0.00 3.00 0.00 28.00 18.67 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 20.00 406.00 221.00 12536.00 2044.00 46.51 69.67 72.59 10.52 186.61 1.57 98.40
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 5.00 0.00 510.00 0.00 3996.00 15.67 144.60 290.53 0.00 290.53 1.96 100.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 0.00 0.00 510.00 0.00 3508.00 13.76 144.06 282.85 0.00 282.85 1.96 100.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 0.00 0.00 510.00 0.00 4076.00 15.98 138.05 282.61 0.00 282.61 1.96 100.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 6.00 44.00 466.00 1272.00 3808.00 19.92 143.29 268.35 79.00 286.22 1.96 100.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 6.00 0.00 4.00 0.00 40.00 20.00 0.00 1.00 0.00 1.00 1.00 0.40
xvdf 0.00 0.00 17.00 493.00 524.00 3484.00 15.72 145.42 279.91 86.82 286.56 1.96 100.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 0.00 24.00 486.00 544.00 2292.00 11.12 146.01 290.81 73.17 301.56 1.96 100.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 5.00 0.00 510.00 0.00 3264.00 12.80 144.30 284.31 0.00 284.31 1.96 100.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 0.00 43.00 467.00 1104.00 2980.00 16.02 149.16 290.46 114.79 306.64 1.96 100.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 0.00 0.00 510.00 0.00 2948.00 11.56 144.12 281.62 0.00 281.62 1.96 100.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 5.00 0.00 4.00 0.00 36.00 18.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 11.00 195.00 366.00 5828.00 2860.00 30.97 84.44 189.44 13.62 283.11 1.78 99.60
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 2.00 0.00 1.00 0.00 12.00 24.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 46.00 660.00 32.00 20032.00 820.00 60.27 5.38 7.92 7.76 11.38 1.41 97.60
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 34.00 683.00 25.00 22400.00 652.00 65.12 4.54 6.41 6.33 8.64 1.38 98.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 37.00 684.00 30.00 21784.00 764.00 63.16 5.07 7.05 6.92 10.13 1.39 99.20
ubuntu@ip-10-28-170-37:~$
I am seeing %util reaching close to 100%. What does load > 2.0 implies and will moving data to a bigger instance could solve my problem. or it can be done with out increasing my instance size. (Current instance size is m1.large)
Best Answer
Your load should be lower than the amount of CPU-cores your server has. So a sixteen core machine should have load below 16.0. Load 1.00 means 1 CPU core is used 100%.
Iostat shows the disk usage. So moving over to a bigger instance will only solve the issue when there are more iops available on that instance. I think the AWS High IO instances probably will help more.
Maybe this article can help you abit; http://blog.joshsoftware.com/2012/11/22/amazon-ec2-mongodb-configuration-great-performance/