FTP Rates drop off after a short time

bandwidthftp

I am trying to transfer a number of large (10's of GB up to 100's of GB) files to an HP DL380 host, running vsftpd on a 10 GigE network / LAN. The transfer initially starts out fast, i.e. 45-60 MB/sec or roughly the speed of the disks. After several minutes (usually about 10-12) the transfer rate drops to a trickle (250 KB/sec or so). We do this sort of thing all the time to other servers, so it seems there is something with the server itself or perhaps a firewall between the client / server.

I am trying to troubleshoot this issue, and why our rates would suddenly and dramatically change. It does not seem to have anything to do with load on the network or the client/server. We suspected it had something to do with PASSIVE/ACTIVE mode, but that hasn't played out (transfers are in PASSIVE mode).

I'm hoping someone out there has seen something similar, i.e. a dramatic drop off in FTP performance. Some network parameter perhaps? A config item in vsftpd?

Anyway, any suggestions appreciated.

Best Answer

Before zeroing in on the FTP aspect, you should perform some synthetic tests to rule out other causes. (I've assumed both the client and the server are Linux since you are running vsftpd.)

  • Network packet drops: Check the output of cat /proc/net/dev on the client and server to see if there are any errors or dropped packets.
  • Network load testing: Use a throughput testing tool such as iperf with a long test time to see if you can reproduce the issue. Monitor throughput with cat /proc/net/dev.
  • MTU: If you're using 10GbE, you may have jumbo frames enabled (that is, MTU size greater than 1500). If you do, make sure that all the hosts and network devices support jumbo frames and that you aren't filtering ICMP packets anywhere (ICMP packets are used for MTU path discovery).
  • Server disk: On the server, use dd to read from /dev/zero and write to a local file on the target filesystem. Monitor throughput with iostat.
  • Client disk: May not need to test this given that you can FTP successfully to other servers from this client.

If any of the above tests show problems, you should have a clue about where to look next. Otherwise, look into the FTP side:

  • If a transfer has slowed down, what happens if you disconnect and then resume the transfer? Is it slow straight away, or does it go fast again before slowing down?
  • Do you get different behaviour when initiating the transfer from the client side vs the server side (i.e. connecting to an FTP server on the client)?
  • Check the memory usage of the FTP server and client using top. If there's a memory leak something may be swapping and slowing everything down.
  • Check the CPU usage of the FTP server and client for any strange behaviour.