Linux VM too slow (vmware) with a lot of resources available

linuxperformanceredhatvmware-workstation

I got a problem with my VM and I ran out of imagination on what can be causing it.

My VM is Red Hat Enterprise Linux Server release 6.9 (Santiago) 

Kernel: 2.6.32-696.30.1.el6.x86_64

These are my VM specs:
6 CPUS:

model name      : Intel(R) Xeon(R) CPU E7- 2850  @ 2.00GHz
model name      : Intel(R) Xeon(R) CPU E7- 2850  @ 2.00GHz
model name      : Intel(R) Xeon(R) CPU E7- 2850  @ 2.00GHz
model name      : Intel(R) Xeon(R) CPU E7- 2850  @ 2.00GHz
model name      : Intel(R) Xeon(R) CPU E7- 2850  @ 2.00GHz
model name      : Intel(R) Xeon(R) CPU E7- 2850  @ 2.00GHz

47G of RAM.

Currently, CPU is looking good, I haven't loaded all my apps yet.

    Cpu0  :  0.8%us,  2.1%sy,  0.0%ni, 96.8%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st
    Cpu1  :  0.6%us,  1.2%sy,  0.0%ni, 97.9%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st
    Cpu2  :  0.8%us,  1.1%sy,  0.0%ni, 97.7%id,  0.4%wa,  0.0%hi,  0.0%si,  0.0%st
    Cpu3  :  0.4%us,  0.6%sy,  0.0%ni, 93.2%id,  0.1%wa,  0.0%hi,  5.6%si,  0.0%st
    Cpu4  :  0.7%us,  2.3%sy,  0.0%ni, 96.6%id,  0.4%wa,  0.0%hi,  0.0%si,  0.0%st
    Cpu5  :  0.6%us,  0.8%sy,  0.0%ni, 98.3%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st
    Mem:  49422016k total,  6087036k used, 43334980k free,    51044k buffers
    Swap: 16777208k total,        0k used, 16777208k free,  1639364k cached

Memory is good too:

                 total       used       free     shared    buffers     cached
    Mem:           47G       5.8G        41G       228K        50M       1.6G
    -/+ buffers/cache:       4.2G        42G
    Swap:          15G         0B        15G

I have been running top, vmstat, sar, iotop, and those stats look good, but the VM is quite slow, even a simple yum search takes forever, to the point that cntl+C has a hard time trying to terminate the session.

Here some vmstat (5sec)
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 43174780  51656 1794176    0    0    11    20  150  192  1  2 97  0  0
 5  0      0 43167188  51696 1800668    0    0   586   928 1012 1043  2  3 92  3  0
 1  0      0 43157116  51728 1809716    0    0   947  2144  854 1021  0  3 94  3  0
 4  0      0 43147792  51736 1819384    0    0   947  2693 1304 1087  1  3 93  3  0
 1  0      0 43138252  51752 1828816    0    0   949    16  786 1046  0  3 95  2  0

If you see, the server is pretty much idle, but for some strange reason things are taking forever to complete.

For example, a yum search nc

real    0m52.324s
user    0m12.564s
sys     0m34.598s

While my other VM takes 2 seconds.

real    0m2.336s
user    0m0.871s
sys     0m0.394s

Any help is greatly appreciated!

Best Answer

The behavior you describe is often due to an overloaded disk subsystem. The uninterruptible yum update is a good indication that some processes (yum in this case) are in D state (deep sleep), meaning they are waiting for disks to return/accept data.

For additional diagnostic, please provide the output of iostat -x -k 1 while executing some command you find particular slow.

Related Topic