Linux – Apache2 benchmarks – very poor performance

apache-2.2linuxperformance

I have two servers on which I test the configuration of apache2.

The first server: 

4GB of RAM, AMD Athlon (tm) 64 X2 Dual Core Processor 5600 + 
Apache 2.2.3, mod_php, mpm prefork: 
Settings: 

Timeout 100 
KeepAlive On 
MaxKeepAliveRequests 150 
KeepAliveTimeout 4 

<IfModule Mpm_prefork_module> 
    StartServers 7 
    MinSpareServers 15 
    MaxSpareServers 30 
    MaxClients 250 
    MaxRequestsPerChild 2000 
</ IfModule> 

Compiled in modules: 
  core.c 
  mod_log_config.c 
  mod_logio.c 
  prefork.c 
  http_core.c 
  mod_so.c

Second server: 

8GB of RAM, Intel (R) Core (TM) i7 CPU 920@2.67GHz 
Apache 2.2.9, **fcgid, mpm worker, suexec**

PHP scripts are running via fcgi-wrapper 

Settings: 

Timeout 100 
KeepAlive On 
MaxKeepAliveRequests 100 
KeepAliveTimeout 4 

<IfModule Mpm_worker_module> 
    StartServers 10 
    MaxClients 200 
    MinSpareThreads 25 
    MaxSpareThreads 75 
    ThreadsPerChild 25 
    MaxRequestsPerChild 1000 
</ IfModule> 

Compiled in modules: 
  core.c 
  mod_log_config.c 
  mod_logio.c 
  worker.c 
  http_core.c 
  mod_so.c

The following test results, which are very strange!

New server (dynamic content – php via fcgid+suexec):

Server Software:    Apache/2.2.9
Server Hostname:    XXXXXXXX
Server Port:    80
Document Path:  XXXXXXX
Document Length:    179512 bytes
Concurrency Level:  10
Time taken for tests:   0.26276 seconds
Complete requests:  1000
Failed requests:    0
Total transferred:  179935000 bytes
HTML transferred:   179512000 bytes
Requests per second:    38.06
Transfer rate:  6847.88 kb/s received
Connnection Times (ms)
    min avg max
Connect:    2   4   54
Processing: 161 257 449
Total:  163 261 503

Old server (dynamic content – mod_php):

Server Software:    Apache/2.2.3
Server Hostname:    XXXXXX
Server Port:    80
Document Path:  XXXXXX
Document Length:    187537 bytes
Concurrency Level:  10
Time taken for tests:   173.073 seconds
Complete requests:  1000
Failed requests:    22
(Connect: 0, Length: 22, Exceptions: 0)
Total transferred:  188003372 bytes
HTML transferred:   187546372 bytes
Requests per second:    5777.91
Transfer rate:  1086267.40 kb/s received
Connnection Times (ms)
    min avg max
Connect:    3   3   28
Processing: 298 1724    26615
Total:  301 1727    26643

Old server: Static content (jpg file)

Server Software:        Apache/2.2.3
Server Hostname:        xxxxxxxxx
Server Port:            80

Document Path:          /images/top2.gif
Document Length:        40486 bytes

Concurrency Level:      100
Time taken for tests:   3.558 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      40864400 bytes
HTML transferred:       40557482 bytes
Requests per second:    281.09 [#/sec] (mean)
Time per request:       355.753 [ms] (mean)
Time per request:       3.558 [ms] (mean, across all concurrent requests)
Transfer rate:          11217.51 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        3   11   4.5     12      23
Processing:    40  329  61.4    339    1009
Waiting:        6  282  55.2    293     737
Total:         43  340  63.0    351    1020

New server – static content (jpg file)

Server Software:        Apache/2.2.9
Server Hostname:        XXXXX
Server Port:            80

Document Path:          /images/top2.gif
Document Length:        40486 bytes

Concurrency Level:      100
Time taken for tests:   3.571531 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      41282792 bytes
HTML transferred:       41030080 bytes
Requests per second:    279.99 [#/sec] (mean)
Time per request:       357.153 [ms] (mean)
Time per request:       3.572 [ms] (mean, across all concurrent requests)
Transfer rate:          11287.88 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        2   63  24.8     66     119
Processing:   124  278  31.8    282     391
Waiting:        3   70  28.5     66     164
Total:        126  341  35.9    350     443

I noticed that in the apache error.log is a lot of entries:

[notice] mod_fcgid: call /www/XXXXX/public_html/forum/index.php with wrapper /www/php-fcgi-scripts/XXXXXX/php-fcgi-starter

What I have omitted, or do not understand?

Such a difference in requests per second? Is it possible? What could be the cause?

Best Answer

The thing that jumps out at me is worker vs prefork. To isolate, you should try running prefork on the "new" / "second" server.

I'm also skeptical of the "old" benchmark:

Transfer rate:  1086267.40 kb/s received

that is equal to 1.03 GB/s, or 8.2 Gbps. Obviously, that is faster than a 1 Gbps port. Are you running the same test here, or benchmarking the old locally and the new remotely?

Hopefully this is not too basic of a question, but are you benching the same load/application? Eg, if you were to compare loading a static file, like an image, to a php page that is doing some processing, you're going to get different results. If your php page is also touching a database, and the database is different on the two servers, that's also going to affect the results.

I would suggest -- again, in the interest of isolation -- benchmarking static files between the two. If you don't see a difference there, try the same simple php page on both (that is not part of another app, just doing something simple like hello world in a loop).

Once you've done an apples vs apples benchmark on the various pieces, you should have enough information to isolate down what is causing the difference (is it your database server? the file system? the apache forking model? the apache version? the network? the benchmark itself?).

Related Topic