Freebsd – Web site kills hard disk I/O, how to prevent

ddosfreebsdhard drive

The situation: I have a server, on which we have 2-3 projects. Starting not long ago, the server started hanging up (We could not connect to it by ssh, and the connected clients had to wait 20 minutes for top to give results)

Early today I managed to execute gstat while it was in this state and saw, that it stays on 100% on da0, da0s1 and da0s1f. I dont quite know what those ids meen, but I understand that some processes just kill the HD by bombing it down with requests.

I ask of some propositions. I dont know how to find the culpit and can't prevent this.

I have freebsd on server.

Best Answer

If your version of FreeBSD is relatively modern, top has a -m option that shows the top I/O talkers if you supply it with the "io" parameter:

top -m io

In this case, I'd also use the -S option (to show system processes, in case one of them is the culprit). To behave better under load, I would use -q (to renice it to run at a higher priority), and -u (to skip reading /etc/passwd, which should help it load faster).

Since it's taking so long to run top, I'd tell it to display just two passes of its output (-d 2), and then run in batch mode (-b), so it will automatically exit.

The first moment that you run top in this way, its first section of output will show cumulative I/O counts for a number of process for quite a ways back (maybe since boot time? I'm actually not sure about this). In the first display, you can see who your top talkers have been over time. In the second display, you can see your top talkers in the past two seconds.

So, putting it all together, and running a find so that some actual I/O is happening:

# top -S -m io -qu -b -d 2 10
last pid: 39560;  load averages:  0.28,  0.19,  0.08  up 6+04:02:29    11:28:28
125 processes: 2 running, 104 sleeping, 19 waiting

Mem: 96M Active, 668M Inact, 122M Wired, 25M Cache, 104M Buf, 17M Free
Swap: 2048M Total, 96K Used, 2048M Free


  PID    UID     VCSW  IVCSW   READ  WRITE  FAULT  TOTAL PERCENT COMMAND
   11      0        0 81032823      0      0      0      0   0.00% idle: cpu0
39554    105   129857 556534  74894      0      0  74894  13.62% find
39533    105   443603 614796      0      0      0      0   0.00% sshd
   36      0   1793393      0      0      0      0      0   0.00% irq23: vr0
   24      0   2377710   2680      0      0      0      0   0.00% irq20: atapci0
   50      0   533513 3415672     66 345350      0 345416  62.81% syncer
   13      0   78651569   7230      0      0      0      0   0.00% swi4: clock sio
    5      0   1911601  20905      0      0      0      0   0.00% g_down
    4      0   2368511  12100      0      0      0      0   0.00% g_up
   37      0    53308    313      0      0      0      0   0.00% acpi_thermal

last pid: 39560;  load averages:  0.28,  0.19,  0.08  up 6+04:02:31    11:28:30
125 processes: 2 running, 104 sleeping, 19 waiting
CPU:  1.9% user,  0.0% nice,  6.0% system,  2.2% interrupt, 89.9% idle
Mem: 96M Active, 671M Inact, 123M Wired, 25M Cache, 104M Buf, 14M Free
Swap: 2048M Total, 96K Used, 2048M Free

  PID    UID     VCSW  IVCSW   READ  WRITE  FAULT  TOTAL PERCENT COMMAND
   11      0        0   1115      0      0      0      0   0.00% idle: cpu0
39554    105      606    651    501      0      0    501 100.00% find
39533    105      616    695      0      0      0      0   0.00% sshd
   36      0     1251      0      0      0      0      0   0.00% irq23: vr0
   24      0      501     20      0      0      0      0   0.00% irq20: atapci0
   50      0        2      2      0      0      0      0   0.00% syncer
   13      0      313      3      0      0      0      0   0.00% swi4: clock sio
    5      0      501     26      0      0      0      0   0.00% g_down
    4      0      501      8      0      0      0      0   0.00% g_up
   37      0        0      0      0      0      0      0   0.00% acpi_thermal

Once you narrow down which process is doing all of the I/O, you can use truss or the devel/strace or sysutils/lsof ports to see what your disk-hungry processes are doing. (if your system is very busy, of course, you won't be able to install the ports easily):

For example, to see what files and other resources my ntpd process is using:

# lsof -p 890
ntpd        890   root  cwd     VDIR       0,93       1024        2 /
ntpd        890   root  rtd     VDIR       0,93       1024        2 /
ntpd        890   root  txt     VREG       0,98     340940   894988 /usr/sbin/ntpd
ntpd        890   root  txt     VREG       0,93     189184    37058 /libexec/ld-elf.so.1
ntpd        890   root  txt     VREG       0,93      92788    25126 /lib/libm.so.5
ntpd        890   root  txt     VREG       0,93      60060    25130 /lib/libmd.so.4
ntpd        890   root  txt     VREG       0,98      16604   730227 /usr/lib/librt.so.1
ntpd        890   root  txt     VREG       0,93    1423460    25098 /lib/libcrypto.so.5
ntpd        890   root  txt     VREG       0,93    1068216    24811 /lib/libc.so.7
ntpd        890   root    0u    VCHR       0,29        0t0       29 /dev/null
ntpd        890   root    1u    VCHR       0,29        0t0       29 /dev/null
ntpd        890   root    2u    VCHR       0,29        0t0       29 /dev/null
ntpd        890   root    3u    unix 0xc46da680        0t0          ->0xc4595820
ntpd        890   root    5u    PIPE 0xc4465244          0          ->0xc446518c
ntpd        890   root   20u    IPv4 0xc4599190        0t0      UDP *:ntp
ntpd        890   root   21u    IPv6 0xc4599180        0t0      UDP *:ntp
ntpd        890   root   22u    IPv4 0xc4599400        0t0      UDP heffalump.prv.tycho.org:ntp
ntpd        890   root   23u    IPv4 0xc4599220        0t0      UDP ns0.prv.tycho.org:ntp
ntpd        890   root   24u    IPv4 0xc45995c0        0t0      UDP imap.prv.tycho.org:ntp
ntpd        890   root   25u    IPv6 0xc4599530        0t0      UDP [fe80:4::1]:ntp
ntpd        890   root   26u    IPv6 0xc45993b0        0t0      UDP localhost:ntp
ntpd        890   root   27u    IPv4 0xc4599160        0t0      UDP localhost:ntp
ntpd        890   root   28u     rte 0xc42939b0        0t0

... and what system calls it's making (note that this can be resource-intensive):

# truss -p 890
SIGNAL 17 (SIGSTOP)
select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) ERR#4 'Interrupted system call'
SIGNAL 14 (SIGALRM)
sigreturn(0xbfbfea10,0xe,0x10003,0xbfbfea10,0x0,0x806aed0) ERR#4 'Interrupted system call'
select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) ERR#4 'Interrupted system call'
SIGNAL 14 (SIGALRM)
sigreturn(0xbfbfea10,0xe,0x10003,0xbfbfea10,0x0,0x806aed0) ERR#4 'Interrupted system call'
select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) ERR#4 'Interrupted system call'
SIGNAL 14 (SIGALRM)
sigreturn(0xbfbfea10,0xe,0x10003,0xbfbfea10,0x0,0x806aed0) ERR#4 'Interrupted system call'
^C

sysutils/strace is similar to truss, but you'll need to have the /proc filesystem mounted:

# strace -p 890
strace: open("/proc/...", ...): No such file or directory
trouble opening proc file

# grep ^proc /etc/fstab
proc            /proc                           procfs          rw,noauto       0       0

# mount /proc

# mount | grep /proc
procfs on /proc (procfs, local)

... and then it will work:

# strace -p 890
Process 890 attached - interrupt to quit
--- SIGALRM (Alarm clock: 14) ---
--- SIGALRM (Alarm clock: 14) ---
syscall_417(0xbfbfea10)                 = -1 (errno 4)
select(29, [?], NULL, NULL, NULL)       = -1 EINTR (Interrupted system call)
--- SIGALRM (Alarm clock: 14) ---
--- SIGALRM (Alarm clock: 14) ---
syscall_417(0xbfbfea10)                 = -1 (errno 4)
select(29, [?], NULL, NULL, NULL^C <unfinished ...>
Process 890 detached

Good luck - let us know what you discover! Once you have the process(es) identified, I may be able to assist further.

EDIT: Note that running lsof , truss and strace can themselves be intensive. I've done some minor updates to try to reduce their impact. Also, if a process is spawing many children quickly, you may have to tell truss or strace to follow child processes with the -f argument.

Related Topic