I have a 512 MB VPS and hosting 2 WordPress websites on this CentOS server. I have installed apache + mysql + PHP + fast cgi on this server. Everything was working great from last 6 months. I have 500 users per day on both websites combined. So no huge load.
But from last night (it's been 12 hours) I have 100% CPU usage and high memory usage. The websites as well as server is not accessible. I tried to reboot server thought it could just be some error but nothing worked.
It's the output of top, but I could not understand what is the problem and how could I fix it. Seems like there are tons of php-cgi & httpd processed.
top - 09:11:43 up 2 min, 1 user, load average: 26.91, 10.07, 3.67
Tasks: 137 total, 28 running, 109 sleeping, 0 stopped, 0 zombie
Cpu(s): 36.4%us, 57.5%sy, 0.0%ni, 4.1%id, 1.4%wa, 0.0%hi, 0.1%si, 0.6%st
Mem: 511036k total, 505416k used, 5620k free, 3280k buffers
Swap: 0k total, 0k used, 0k free, 12240k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
804 root 20 0 36160 540 4 S 11.7 0.1 0:03.69 rsyslogd
1548 robert 20 0 51656 25m 812 R 5.8 5.1 0:00.76 php-cgi
1549 robert 20 0 50412 23m 528 R 5.8 4.8 0:00.70 php-cgi
1552 robert 20 0 50704 24m 764 R 5.8 4.9 0:00.64 php-cgi
1568 robert 20 0 44940 18m 760 R 5.8 3.7 0:00.42 php-cgi
1573 robert 20 0 38680 12m 792 R 5.8 2.6 0:00.32 php-cgi
1584 robert 20 0 31964 6300 704 R 5.8 1.2 0:00.19 php-cgi
1553 robert 20 0 49544 23m 1184 R 4.4 4.7 0:00.61 php-cgi
1554 robert 20 0 49544 23m 972 R 4.4 4.7 0:00.60 php-cgi
1557 robert 20 0 46288 19m 816 R 4.4 4.0 0:00.57 php-cgi
1558 robert 20 0 46288 19m 836 R 4.4 4.0 0:00.52 php-cgi
1563 robert 20 0 45452 19m 1104 R 4.4 3.9 0:00.49 php-cgi
1564 robert 20 0 45452 19m 1136 R 4.4 3.9 0:00.46 php-cgi
1565 robert 20 0 44948 18m 764 R 4.4 3.7 0:00.43 php-cgi
1569 robert 20 0 35492 9872 768 R 4.4 1.9 0:00.39 php-cgi
1572 robert 20 0 38680 12m 816 R 4.4 2.6 0:00.34 php-cgi
1574 robert 20 0 38376 12m 784 R 4.4 2.5 0:00.30 php-cgi
1576 robert 20 0 38388 12m 800 R 4.4 2.5 0:00.26 php-cgi
1583 robert 20 0 32736 7688 1332 R 4.4 1.5 0:00.20 php-cgi
1585 robert 20 0 31312 5832 1032 R 4.4 1.1 0:00.17 php-cgi
1586 robert 20 0 31312 5856 1012 R 4.4 1.1 0:00.14 php-cgi
1589 robert 20 0 30008 5320 1728 R 4.4 1.0 0:00.12 php-cgi
1593 robert 20 0 30012 5208 1620 R 4.4 1.0 0:00.07 php-cgi
1594 robert 20 0 30016 5156 1616 R 4.4 1.0 0:00.07 php-cgi
1595 robert 20 0 30008 5320 1728 D 4.4 1.0 0:00.07 php-cgi
1597 robert 20 0 12072 464 276 R 4.4 0.1 0:00.03 php-cgi
1579 robert 20 0 32736 7844 1444 R 2.9 1.5 0:00.24 php-cgi
28 root 20 0 0 0 0 S 1.5 0.0 0:02.65 kswapd0
991 mysql 20 0 139m 14m 692 S 1.5 2.8 0:04.41 mysqld
1186 robert 20 0 35172 6184 984 R 1.5 1.2 0:00.07 httpd
1546 robert 20 0 53412 28m 1632 S 1.5 5.6 0:00.75 php-cgi
1596 robert 20 0 2696 476 228 R 1.5 0.1 0:00.01 top
1 root 20 0 2900 200 4 S 0.0 0.0 0:00.77 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.13 watchdog/0
7 root 20 0 0 0 0 S 0.0 0.0 0:00.33 events/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cgroup
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khelper
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 netns
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 async/mgr
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pm
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 sync_supers
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 bdi-default
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/0
16 root 20 0 0 0 0 R 0.0 0.0 0:03.46 kblockd/0
17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpid
18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpi_notify
19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpi_hotplug
20 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/0
21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata_aux
22 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksuspend_usbd
23 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khubd
24 root 20 0 0 0 0 S 0.0 0.0 0:00.03 kseriod
25 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/0
26 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/0
27 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khungtaskd
29 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd
30 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/0
31 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/0
36 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthrotld/0
38 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kpsmoused
39 root 20 0 0 0 0 S 0.0 0.0 0:00.00 usbhid_resumer
189 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0
190 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1
208 root 20 0 0 0 0 S 0.0 0.0 0:00.00 virtio-blk
263 root 20 0 0 0 0 S 0.0 0.0 0:00.03 jbd2/vda-8
264 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ext4-dio-unwrit
333 root 16 -4 2512 400 4 S 0.0 0.1 0:00.15 udevd
361 root 20 0 0 0 0 S 0.0 0.0 0:00.00 virtio-net
364 root 20 0 0 0 0 S 0.0 0.0 0:00.00 vballoon
543 root 18 -2 2508 396 4 S 0.0 0.1 0:00.00 udevd
546 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kstriped
600 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kauditd
816 root 20 0 2020 88 4 S 0.0 0.0 0:00.00 acpid
833 root 20 0 8940 512 4 S 0.0 0.1 0:00.00 sshd
868 root 20 0 3044 184 4 S 0.0 0.0 0:00.00 mysqld_safe
992 root 20 0 0 0 0 S 0.0 0.0 0:00.02 flush-253:0
1084 root 20 0 12960 636 4 S 0.0 0.1 0:00.03 master
1091 postfix 20 0 13036 620 4 S 0.0 0.1 0:00.00 pickup
1092 postfix 20 0 13108 672 4 S 0.0 0.1 0:00.00 qmgr
1094 root 20 0 34900 5036 88 S 0.0 1.0 0:00.52 httpd
1097 robert 20 0 20568 2320 4 S 0.0 0.5 0:00.00 httpd
1103 root 20 0 3956 560 4 S 0.0 0.1 0:00.01 crond
1106 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.05 httpd
1107 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.02 httpd
1119 postfix 20 0 14204 848 4 S 0.0 0.2 0:00.00 smtpd
1134 postfix 20 0 13180 656 4 S 0.0 0.1 0:00.00 cleanup
1138 root 20 0 2008 60 4 S 0.0 0.0 0:00.00 mingetty
1140 root 20 0 2008 56 4 S 0.0 0.0 0:00.00 mingetty
1142 root 20 0 2008 60 4 S 0.0 0.0 0:00.00 mingetty
1144 root 20 0 2008 64 4 S 0.0 0.0 0:00.00 mingetty
1146 root 20 0 2008 64 4 S 0.0 0.0 0:00.00 mingetty
1148 root 20 0 2008 64 4 S 0.0 0.0 0:00.00 mingetty
1150 postfix 20 0 13232 680 4 S 0.0 0.1 0:00.04 smtp
1151 postfix 20 0 13232 672 4 S 0.0 0.1 0:00.00 smtp
1159 root 20 0 11884 740 8 S 0.0 0.1 0:00.01 sshd
1160 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.09 httpd
1164 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.03 httpd
1165 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.03 httpd
1172 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.05 httpd
1174 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.07 httpd
1175 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.07 httpd
1184 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.04 httpd
1185 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.09 httpd
1187 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.05 httpd
1188 robert 20 0 35600 5412 16 S 0.0 1.1 0:00.10 httpd
1189 robert 20 0 35172 5264 16 S 0.0 1.0 0:00.03 httpd
1190 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.02 httpd
1191 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.11 httpd
1196 robert 20 0 11884 748 4 S 0.0 0.1 0:00.00 sshd
1201 robert 20 0 8220 448 4 S 0.0 0.1 0:00.00 sftp-server
1208 robert 20 0 35172 5268 16 S 0.0 1.0 0:00.12 httpd
1214 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.04 httpd
1220 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.02 httpd
1221 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.04 httpd
1222 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.05 httpd
1223 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.02 httpd
1229 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.07 httpd
1238 root 20 0 11884 736 8 S 0.0 0.1 0:00.01 sshd
1260 robert 20 0 12020 752 4 S 0.0 0.1 0:00.59 sshd
1265 robert 20 0 3180 308 4 S 0.0 0.1 0:00.14 bash
1266 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.07 httpd
1286 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.10 httpd
1287 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.02 httpd
1294 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.01 httpd
1295 robert 20 0 35172 5224 16 S 0.0 1.0 0:00.06 httpd
1296 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.02 httpd
1332 robert 20 0 35172 5216 16 S 0.0 1.0 0:00.09 httpd
1509 root 20 0 4324 668 4 S 0.0 0.1 0:00.00 crond
1510 root 20 0 4324 668 4 S 0.0 0.1 0:00.00 crond
1512 robert 20 0 6572 380 4 S 0.0 0.1 0:00.02 wget
1513 robert 20 0 6572 376 4 S 0.0 0.1 0:00.02 wget
1545 robert 20 0 53412 28m 1636 S 0.0 5.6 0:00.76 php-cgi
1547 robert 20 0 53412 28m 1632 S 0.0 5.6 0:00.74 php-cgi
Can you guys look at it. Thanks
EDIT:
I have ton of these entries in access_logs
104.245.97.218 - - [14/Apr/2015:07:54:18 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:18 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:19 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:19 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:20 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:20 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:21 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:21 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:21 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:22 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:22 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:23 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:23 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:24 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:24 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:24 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:25 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:25 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:26 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:26 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:26 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:27 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:27 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:28 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:28 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:29 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:29 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:29 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:30 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:30 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
104.245.97.218 - - [14/Apr/2015:07:54:31 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
Best Answer
It looks like you're getting hit by a pretty standard brute-force password guessing attack against Wordpress. As the linked article says,
So it looks like a responsive ip-specific blocking tool like
fail2banis likely to be the way to go, here. Failing that, this SF question suggests reconfiguring apache to refuse access to that script, which at least returns a 403 Forbidden instead of executing the script - that is much cheaper, computationally, than running it for each request, and would decrease the sever load.Edit: congratulations on having
fail2baninstalled. Sadly, it's not magic pixie dust that automagically blocks all badness, it's a highly-configurable framework for responding to certain classes of entries in logfiles with ip-specific bans viaiptables. You will have to configure an appropriate jail before it can help you.If that doesn't sound fun, you could ban this specific IP and see if it helps, with
(assuming your server is on port 80).