We have been trying to do some performance benchmarking for our webapp written using python that uses mongodb heavily and we have found the following.
We tried using 1st Gen Extra Large ec2 servers with 8 ecu's and 15 GB memory
- The python on ec2 servers is at least 30% slower than the local machine
- The disk I/O is extremely slow. The mongostat and iostat results show that the disk write is around 1MBpS
- The programs themselves run so much slower than the local machine
We have not been able to figure out why all this is happening. The local machines we are talking about have 8GB RAM and i5 processors.
UPDATE
The way we tested python was that we ran a loop that would take 10 seconds to complete with no disk read or write. The same took at least 30% more time on each trial.
Does this relate to this?
Best Answer
There are a number of factors for EC2 machines to be slow. Disks are not attached directly to the instance. Instead the ebs volumes are large network disks and whatever you write to them is sent across the network to these disks. Now usually the latency is quite low but, of course, in comparison to something which is directly attached to your machine it will appear slow.
It is a virtual machine. No matter what you do it has to compete with other machines for CPU cycles. Run top if you are using Linux and check out CPU steal percentage. A non zero number will indicate that there is high competition for CPU. In any case virtual CPUs are not as fast as actual CPUs for comparable Processors.
Another personal observation is that luck plays an important role in EC2 (yes!). At times you get an older hardware which is just not as fast. Another personal experience is that at times you get amd opteron processors which are usually not as fast as Intel based. I am not suggesting that AMD processors are bad but it seems that in this case Intel ones work faster. Maybe they are of newer generation.
Having maintained mongo on EC2, I totally understand your pain. I would suggest that try to keep as much data in-memory as possible. In general, EC2 is not actually designed for vertical scaling. It is beneficial to have a lot of smaller instance dividing work then have a huge instance doing everything alone.