Windows Server 2008 x64, Large File Transfers, and Memory Usage

64-bitfile-transfermemorywindows-server-2008

I'm following up on this:

Copying large file to remote server causes it to run out of physical memory

But figured I'd post this separately.

I have a 64-bit server, Xeon 5405, with 14GB memory, and a Areca 1680 SATA storage controller, that I use for backups, NMS, and testing. It has Intel Pro/1000EB and PL NICs. I have VMWare server 2.0 running on it, and I want to do some P2V conversions so I can test an application's service pack.

I discovered that in the process of copying the 44GB VMDK file to the server that it runs out of memory. I thought it was a memory leak in VMWare Converter, but doing the conversion on a different 32-bit server running Server 2008 doesn't have the problem, as long as I do the conversion to the 32-bit server's local disk. If I then try to copy the 44GB VMDK file to the server via Windows File Sharing, the server eventually gets all of its memory consumed behind the scenes and the file transfer slows to a crawl.

So I tried FTP from the remote server. Unfortunately the Windows FTP client wants to use a Temp file on C:\ for its work, which won't work since I don't have that much free space on C:. So I used Filezilla as my client, and I see similar high memory usage (although it goes up to 75% instead of 99%), and the transfers start out fast, and then gets really slow. I see this fast/slow/fast/slow behaviour in the FTP client's transfer rate numbers, and similar behaviour in the network

This explains a lot of the issue:

http://blogs.technet.com/askperf/archive/2007/05/08/slow-large-file-copy-issues.aspx

I tried ESEUtil and doesn't do the memory thing, but its not very fast.

I'm using TeraCopy at this point. It gets 16MB/sec which is pretty crappy but at least its consistent.

I don't think its my storage controller, as I've copied multi-GB files around on the server locally without a problem.

Wondering if folks smarter than I am know the magic answer that makes this go away. I never used to worry about this stuff until 64-bit Windows came along. I also have a problem on the server I'm making a virtual copy of with SQL Server memory usage, but that seems to have stabilized.

Best Answer

Try using RichCopy 4 by Ken Tamaru.

http://technet.microsoft.com/en-us/magazine/2009.04.utilityspotlight.aspx?pr=blog

Try running with the "Turn off system buffer" option.

HTH.