I have a MediaTemple (dv) 4 server with 512mb running Apache 2.2.21 64-bit.
Within 60 seconds of starting, httpd is spawning 2-4 processes, each consuming ~150mb of memory (according to top's 'RES' column). This very quickly crashes my server. By 'crashes' I mean all urls start timing out, as a result I cannot log into the attached plask panel. Additionally, ssh no longer works (connection times out) and the real-time stats provided by MediaTemple stop working also. So far the only fix I've found is to restart the server. Apache is running PHP (Zend Engine 2.3.0) serving a simple Drupal install.
I've tried a few suggestions on the media temple wiki with no improvement.
- What could be causing httpd to consume this much memory?
- Is this atypical behaviour?
- What can I do to fix this and restore the server to a usable state?
I'm happy to post server details, logs etc as requested.
EDIT: I should have mentioned earlier, this server has been running fine for 4 months or so, and has only started sporadically crashing like this in the past month. I'm hoping to ascertain if this is simply due to increased Drupal loading, or some other (potentially more malicious) issue.
EDIT: As requested, here's the output of running ps_mem.py today.
-bash-3.2$ wget http://www.pixelbeat.org/scripts/ps_mem.py
-bash-3.2$ sudo python ps_mem.py
Private + Shared = RAM used Program
92.0 KiB + 12.0 KiB = 104.0 KiB qmail-clean
96.0 KiB + 14.0 KiB = 110.0 KiB splogger
116.0 KiB + 23.0 KiB = 139.0 KiB init
128.0 KiB + 12.0 KiB = 140.0 KiB qmail-rspawn
124.0 KiB + 16.0 KiB = 140.0 KiB syslogd
132.0 KiB + 12.0 KiB = 144.0 KiB qmail-lspawn
148.0 KiB + 13.0 KiB = 161.0 KiB qmail-send
208.0 KiB + 28.5 KiB = 236.5 KiB dbus-daemon
232.0 KiB + 36.5 KiB = 268.5 KiB xinetd
240.0 KiB + 32.5 KiB = 272.5 KiB mysqld_safe
328.0 KiB + 20.5 KiB = 348.5 KiB udevd
348.0 KiB + 66.0 KiB = 414.0 KiB courierlogger (4)
444.0 KiB + 85.5 KiB = 529.5 KiB bash
480.0 KiB + 50.0 KiB = 530.0 KiB xfs
592.0 KiB + 36.0 KiB = 628.0 KiB crond
544.0 KiB + 114.0 KiB = 658.0 KiB couriertcpd (4)
1.3 MiB + 82.5 KiB = 1.4 MiB sw-cp-serverd
1.2 MiB + 1.1 MiB = 2.3 MiB sshd (3)
3.1 MiB + 205.5 KiB = 3.3 MiB named
3.9 MiB + 48.2 MiB = 52.1 MiB spamd (2)
63.7 MiB + 387.0 KiB = 64.1 MiB mysqld
108.3 MiB + 9.2 MiB = 117.5 MiB httpd (7)
---------------------------------
245.4 MiB
=================================
A top snapshot, showing one httpd service running as root, others running as the apache user. Is this (one root process) typical?
-bash-3.2$ top -b -n 1
top - 17:09:34 up 27 days, 18:04, 1 user, load average: 0.00, 0.00, 0.00
Tasks: 38 total, 1 running, 37 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 946344k total, 581372k used, 364972k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21727 root 15 0 114m 52m 2112 S 0.0 5.6 0:09.17 spamd
21728 popuser 18 0 114m 50m 972 S 0.0 5.5 0:00.00 spamd
28112 apache 18 0 135m 3612 484 S 0.0 0.4 0:00.03 httpd
28110 root 18 0 259m 15m 8284 S 0.0 1.7 0:00.21 httpd
31854 apache 15 0 259m 9368 1492 S 0.0 1.0 0:00.00 httpd
28113 apache 15 0 260m 9400 1508 S 0.0 1.0 0:00.03 httpd
31853 apache 15 0 260m 9368 1484 S 0.0 1.0 0:00.00 httpd
5599 named 18 0 301m 5112 1976 S 0.0 0.5 0:00.51 named
31851 apache 15 0 309m 61m 4004 S 0.0 6.7 0:01.13 httpd
28124 apache 18 0 312m 63m 4996 S 0.0 6.8 0:03.92 httpd
1865 mysql 15 0 358m 70m 7112 S 0.0 7.6 5:52.36 mysqld
5824 root 18 0 3808 560 472 S 0.0 0.1 0:00.00 courierlogger
5834 root 18 0 3808 556 472 S 0.0 0.1 0:00.00 courierlogger
5842 root 18 0 3808 564 472 S 0.0 0.1 0:00.15 courierlogger
5851 root 18 0 3808 556 472 S 0.0 0.1 0:00.00 courierlogger
7255 qmailq 18 0 3812 428 344 S 0.0 0.0 0:00.00 qmail-clean
7252 qmaill 18 0 3816 556 468 S 0.0 0.1 0:00.00 splogger
7253 root 16 0 3856 468 352 S 0.0 0.0 0:00.00 qmail-lspawn
7254 qmailr 15 0 3856 472 356 S 0.0 0.0 0:00.00 qmail-rspawn
7250 qmails 15 0 3864 564 456 S 0.0 0.1 0:00.07 qmail-send
1418 root 15 0 5928 624 500 S 0.0 0.1 0:10.71 syslogd
1 root 18 0 10368 736 620 S 0.0 0.1 0:05.61 init
1695 root 18 0 10784 1300 1060 S 0.0 0.1 0:00.01 mysqld_safe
31869 serverad 15 0 12080 1716 1324 S 0.0 0.2 0:00.02 bash
31976 serverad 15 0 12624 1120 872 R 0.0 0.1 0:00.00 top
1123 root 15 -4 12636 684 356 S 0.0 0.1 0:00.00 udevd
5822 root 17 0 13188 612 476 S 0.0 0.1 0:00.00 couriertcpd
5832 root 15 0 13188 612 476 S 0.0 0.1 0:00.00 couriertcpd
5840 root 15 0 13188 612 476 S 0.0 0.1 0:00.46 couriertcpd
5849 root 18 0 13188 612 476 S 0.0 0.1 0:00.00 couriertcpd
3077 xfs 15 0 20276 1272 792 S 0.0 0.1 0:00.00 xfs
2035 root 15 0 20892 1180 588 S 0.0 0.1 0:04.17 crond
1427 dbus 19 0 21276 700 492 S 0.0 0.1 0:00.00 dbus-daemon
3523 root 15 0 21664 944 712 S 0.0 0.1 0:00.02 xinetd
28213 sw-cp-se 18 0 59512 2728 1420 S 0.0 0.3 0:00.03 sw-cp-serverd
1504 root 15 0 62680 1228 660 S 0.0 0.1 1:54.58 sshd
31866 root 15 0 96624 3704 2852 S 0.0 0.4 0:00.02 sshd
31868 serverad 15 0 96624 1892 1012 S 0.0 0.2 0:00.04 sshd
… and some memory stats according to ps.
-bash-3.2$ ps aux | sort -nk +4 | tail
named 5599 0.0 0.5 308484 5112 ? Ssl Jul24 0:00 /usr/sbin/named -u named -c /etc/named.conf -u named -t /var/named/run-root
apache 28113 0.0 0.9 266364 9400 ? S 05:15 0:00 /usr/sbin/httpd
apache 31853 0.0 0.9 266364 9368 ? S 16:08 0:00 /usr/sbin/httpd
apache 31854 0.0 0.9 266228 9368 ? S 16:09 0:00 /usr/sbin/httpd
root 28110 0.0 1.6 266092 16028 ? Ss 05:15 0:00 /usr/sbin/httpd
popuser 21728 0.0 5.5 117128 52164 ? S Jul07 0:00 spamd child
root 21727 0.0 5.6 117128 53304 ? Ss Jul07 0:09 /usr/bin/spamd --username=popuser --daemonize --nouser-config --helper-home-dir=/var/qmail --max-children 1 --pidfile=/var/run/spamd/spamd_full.pid --socketpath=/tmp/spamd_full.sock
apache 31851 0.0 6.6 317320 63324 ? S 16:08 0:01 /usr/sbin/httpd
apache 28124 0.0 6.8 319536 64604 ? S 05:15 0:03 /usr/sbin/httpd
mysql 1865 0.0 7.6 367604 72376 ? Sl Jul04 5:52 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock --port=3306
Some potentially relevant snippets from httpd.conf (after adjusting prefork settings).
-bash-3.2$ head httpd.conf -n 128 | tail -60
#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 20
#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive Off
#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100
#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 15
##
## Server-Pool Size Regulation (MPM specific)
##
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers 2
MinSpareServers 3
MaxSpareServers 5
ServerLimit 16
MaxClients 16
MaxRequestsPerChild 400000
</IfModule>
# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers 1
MaxClients 50
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
Thanks!
Best Answer
That's not a good way to measure a processes memory usage, as explained in lots of places.
If your system is running out of memory then you've configured it wrongly. Try measuring the memory usage properly and limit the number of servers first, then you can start to tune the installation.
You're not going to get an answer here - there are too many questions and information that would need to be collected. Start by getting a handle on what's really happenning with the memory and the performance of the system read lots, read some more, establish a mechanism for collecting metrics, for preference set up a test environment where you can try out different configurations / loads quickly, then start tuning.