I have a web application that is used nearly constantly by 5 – 10 users. Between 1 and 3 times a day it will grind to a halt.
When this happens I can see the following in TOP:
last pid: 1744; load averages: 15.50, 45.91, 31.79 up 0+01:04:01 14:47:59
357 processes: 1 running, 351 sleeping, 5 waiting
CPU: 0.2% user, 0.0% nice, 2.8% system, 2.4% interrupt, 94.6% idle
Mem: 1612M Active, 180M Inact, 174M Wired, 8012K Cache, 103M Buf, 108K Free
Swap: 4096M Total, 2104M Used, 1992M Free, 51% Inuse, 40M Out
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
1384 www 1 44 0 156M 3256K pfault 0 0:36 0.00% httpd
1477 www 1 44 0 154M 8008K pfault 1 0:35 0.00% httpd
1053 mysql 161 44 0 199M 54108K umtxn 0 0:27 0.00% mysqld
1441 www 1 44 0 154M 0K WAIT 0 0:26 0.00% <httpd>
1521 www 1 44 0 164M 0K sbwait 1 0:20 0.00% <httpd>
1519 www 1 44 0 156M 7832K pfault 0 0:19 0.00% httpd
1520 www 1 44 0 138M 8724K select 1 0:19 0.00% httpd
1527 www 1 46 0 138M 0K accept 0 0:17 0.00% <httpd>
1422 www 1 44 0 138M 3096K pfault 0 0:16 0.00% httpd
1290 root 1 44 0 130M 9060K select 1 0:15 0.00% httpd
1529 www 1 44 0 164M 3740K pfault 1 0:15 0.00% httpd
1532 www 1 44 0 154M 22704K pfault 0 0:15 0.00% httpd
1530 www 1 44 0 164M 0K sbwait 0 0:14 0.00% <httpd>
1525 www 1 44 0 156M 0K sbwait 1 0:14 0.00% <httpd>
1531 www 1 44 0 138M 8168K pfault 1 0:13 0.00% httpd
1528 www 1 44 0 164M 0K sbwait 0 0:13 0.00% <httpd>
1533 www 1 44 0 156M 3252K pfault 1 0:12 0.00% httpd
1534 www 1 44 0 156M 3592K pfault 1 0:12 0.00% httpd
1544 www 1 44 0 156M 4620K pfault 1 0:12 0.00% httpd
1540 www 1 44 0 156M 4340K pfault 1 0:12 0.00% httpd
1543 www 1 44 0 156M 4744K pfault 0 0:11 0.00% httpd
1546 www 1 44 0 154M 11764K pfault 1 0:11 0.00% httpd
1547 www 1 44 0 154M 8176K pfault 0 0:11 0.00% httpd
1545 www 1 44 0 154M 8660K pfault 1 0:11 0.00% httpd
1541 www 1 44 0 156M 0K WAIT 1 0:10 0.00% <httpd>
1558 www 1 44 0 158M 19996K pfault 0 0:10 0.00% httpd
1559 www 1 44 0 154M 10324K pfault 0 0:10 0.00% httpd
1555 www 1 44 0 154M 18796K pfault 1 0:09 0.00% httpd
1569 www 1 44 0 156M 6208K pfault 0 0:09 0.00% httpd
1566 www 1 44 0 154M 10996K pfault 0 0:09 0.00% httpd
1561 www 1 44 0 156M 15724K pfault 1 0:09 0.00% httpd
1562 www 1 44 0 154M 8076K pfault 0 0:09 0.00% httpd
1572 www 1 44 0 154M 11308K pfault 0 0:09 0.00% httpd
1571 www 1 44 0 154M 0K sbwait 0 0:08 0.00% <httpd>
1553 www 1 44 0 154M 0K sbwait 1 0:08 0.00% <httpd>
1567 www 1 44 0 154M 13108K pfault 1 0:08 0.00% httpd
1565 www 1 44 0 152M 17284K pfault 0 0:08 0.00% httpd
1602 www 1 44 0 156M 4232K pfault 1 0:08 0.00% httpd
1548 www 1 44 0 154M 8948K pfault 0 0:08 0.00% httpd
1577 www 1 44 0 156M 15732K pfault 0 0:08 0.00% httpd
1573 www 1 44 0 156M 16244K pfault 0 0:08 0.00% httpd
1554 www 1 44 0 154M 0K sbwait 0 0:08 0.00% <httpd>
1585 www 1 44 0 156M 0K WAIT 1 0:08 0.00% <httpd>
1552 www 1 44 0 154M 0K sbwait 0 0:08 0.00% <httpd>
1575 www 1 44 0 154M 11956K pfault 0 0:08 0.00% httpd
1570 www 1 44 0 156M 16260K pfault 1 0:08 0.00% httpd
1583 www 1 44 0 156M 15368K pfault 1 0:08 0.00% httpd
1580 www 1 44 0 154M 11112K pfault 0 0:08 0.00% httpd
1600 www 1 44 0 156M 4248K pfault 1 0:08 0.00% httpd
1589 www 1 44 0 154M 0K sbwait 1 0:08 0.00% <httpd>
1578 www 1 44 0 154M 0K sbwait 0 0:08 0.00% <httpd>
1557 www 1 44 0 154M 0K sbwait 1 0:07 0.00% <httpd>
1617 www 1 44 0 154M 11244K pfault 1 0:07 0.00% httpd
1574 www 1 44 0 154M 8360K pfault 1 0:07 0.00% httpd
1560 www 1 44 0 154M 10156K pfault 0 0:07 0.00% httpd
1603 www 1 44 0 156M 4968K pfault 0 0:07 0.00% httpd
1590 www 1 44 0 154M 11860K pfault 0 0:07 0.00% httpd
1592 www 1 44 0 156M 14280K pfault 1 0:07 0.00% httpd
1581 www 1 44 0 156M 6412K pfault 0 0:07 0.00% httpd
1586 www 1 44 0 154M 11176K pfault 0 0:07 0.00% httpd
1608 www 1 44 0 156M 7256K pfault 1 0:07 0.00% httpd
1568 www 1 44 0 154M 10784K pfault 1 0:07 0.00% httpd
1591 www 1 44 0 154M 13256K pfault 0 0:07 0.00% httpd
1610 www 1 44 0 156M 3440K pfault 0 0:07 0.00% httpd
1650 www 1 44 0 156M 0K sbwait 1 0:07 0.00% <httpd>
1616 www 1 44 0 154M 13724K pfault 0 0:07 0.00% httpd
1611 www 1 44 0 154M 10084K pfault 0 0:06 0.00% httpd
1564 www 1 44 0 154M 12220K pfault 1 0:06 0.00% httpd
1607 www 1 44 0 156M 0K sbwait 0 0:06 0.00% <httpd>
1601 www 1 44 0 154M 13728K pfault 1 0:06 0.00% httpd
1634 www 1 44 0 156M 0K sbwait 0 0:06 0.00% <httpd>
1606 www 1 44 0 156M 0K sbwait 1 0:06 0.00% <httpd>
1626 www 1 44 0 156M 3340K pfault 1 0:06 0.00% httpd
1633 www 1 44 0 156M 0K sbwait 0 0:06 0.00% <httpd>
1579 www 1 44 0 154M 8508K pfault 1 0:06 0.00% httpd
1596 www 1 44 0 156M 0K sbwait 1 0:06 0.00% <httpd>
1584 www 1 44 0 156M 2388K pfault 1 0:06 0.00% httpd
1641 www 1 44 0 156M 3060K pfault 0 0:06 0.00% httpd
1632 www 1 44 0 156M 3144K pfault 1 0:06 0.00% httpd
1657 www 1 44 0 156M 0K sbwait 1 0:06 0.00% <httpd>
1630 www 1 44 0 156M 6872K pfault 0 0:06 0.00% httpd
1609 www 1 44 0 156M 0K sbwait 1 0:06 0.00% <httpd>
1661 www 1 44 0 156M 5688K pfault 0 0:05 0.00% httpd
1612 www 1 44 0 156M 17816K sbwait 1 0:05 0.00% httpd
1670 www 1 44 0 156M 4904K pfault 0 0:05 0.00% httpd
1576 www 1 44 0 154M 13980K pfault 1 0:05 0.00% httpd
1673 www 1 44 0 156M 0K sbwait 0 0:05 0.00% <httpd>
1645 www 1 44 0 156M 3312K pfault 0 0:05 0.00% httpd
1597 www 1 44 0 156M 6004K pfault 1 0:05 0.00% httpd
1587 www 1 44 0 156M 6008K sbwait 1 0:05 0.00% httpd
1628 www 1 44 0 156M 0K sbwait 0 0:05 0.00% <httpd>
1660 www 1 44 0 156M 6180K pfault 0 0:05 0.00% httpd
1647 www 1 44 0 158M 0K sbwait 1 0:05 0.00% <httpd>
1582 www 1 44 0 156M 0K sbwait 0 0:05 0.00% <httpd>
1664 www 1 44 0 156M 0K sbwait 0 0:05 0.00% <httpd>
1651 www 1 44 0 156M 0K WAIT 1 0:05 0.00% <httpd>
1674 www 1 44 0 156M 7560K pfault 0 0:05 0.00% httpd
1631 www 1 44 0 156M 6464K pfault 0 0:05 0.00% httpd
1655 www 1 44 0 158M 0K sbwait 0 0:05 0.00% <httpd>
1669 www 1 44 0 156M 0K sbwait 0 0:04 0.00% <httpd>
1659 www 1 44 0 156M 5936K pfault 1 0:04 0.00% httpd
1663 www 1 44 0 156M 5828K pfault 0 0:04 0.00% httpd
1672 www 1 44 0 156M 5764K pfault 0 0:04 0.00% httpd
1613 www 1 44 0 156M 5784K pfault 1 0:04 0.00% httpd
1667 www 1 44 0 156M 3696K pfault 1 0:04 0.00% httpd
1662 www 1 44 0 156M 3336K pfault 0 0:04 0.00% httpd
1671 www 1 44 0 156M 5036K sbwait 0 0:04 0.00% httpd
1675 www 1 44 0 156M 6228K pfault 1 0:04 0.00% httpd
1629 www 1 44 0 156M 5788K pfault 0 0:04 0.00% httpd
Is there any way to find what the httpd processes are getting stuck doing?
Best Answer
First, as ETL wrote, enable
mod_status
withExtendedStatus On
to see current requests and their states, execution times... etc. You can also use anapachetop
to get information which file is most popular. If it's not enough information to you, as timmeyh wrote, you can usestrace
but I will suggest-ff
and-p
switches (make sure that you specified main apache proccess id).-ff
switch-p
switch