I have a very weird problem with few of my servers. It seems that users on these servers, whenever they download from them, either via FTP (proftpd) or HTTP (apache2), big files (over 1GB), the download connection is dropped exactly when they reach 1GB of downloaded data.
The problem is not random, it happens every time, with all accounts, and exactly around the 1GB mark. I cannot figure out what can cause this.
Servers are running Ubuntu Linux Oneiric Ocelot
Best Answer
I have downloaded files much larger than 1 GB (>100 GB even) through ftp, ssh, http and never encountered such a problem in such a consistent manner. But have observed and have been troubleshooting the occasional suspicious tcp-rst packets in other situations. A tcp-rst should not happen in a normal situation, it's almost always an abnormal event that deserves some healthy suspicion.
The fact it's so consistent and the connection is reset as opposed to properly closed is an indication something fishy is going on in my opinion. Firewalls and routers (badly configured ones, or just broken) can be doing that kinda thing. You could run tcpdump whilst such a download is happening and study the output.
My guess is that your ISP may be causing this, unwillingly perhaps due to a broken router or firewall.
At times I have been trying to find out about some mysterious tcp resets myself and used tcpdump for this:
I'd say, around the 980 MB mark, tcpdump everything to a file:
Then use tcpdump to find the reset packet(s):
Then once again use tcpdump and grep to look around the timestamp for anything happening that might give a clue. The -A and -B options will output 200 lines before and after the grepped line.
Ar the servers where this happens using a different ISP? Or a separate router or firewall? Can you verify this does not happen on other servers that use the same ISP, router and firewall?