Windows Server 2008 – Fixing Slow Download from Share on Mixed Ethernet Speeds

file-servergigabit-ethernetnetworkingperformancewindows-server-2008

I have a Windows Server 2008 R2 x64 server that is AD controller and file server. I have a problem that Windows XP clients experience terribly slow (less than 10Mbps, really less than ten-megabits-per-second) downloading of files from a share.

Server is connected to 1Gbps switch using 1Gbps Nvidia NForce card and client is connected using 100Mbps built-in card.

This slow downloading can also be seen when I've booted client computer from CentOS Linux 5.5 Live-USB and used smbclient for downloading. But downloading from a Samba share on Linux server, which is also connected using 1Gbps link is fast.

What is very strange I've created a pair of programs (attached below) which test plain TCP throughput in C#, and they're performing as expected — at about 89Mbps.

I've disabled firewall on client and I'm using dot_nc_l 21000 > NIL on client and dot_nc [client_ip] < 100m.dat on Windows server. And I get about 9 seconds, when copying the same 100MB file from share takes over 2 minutes.

How to eliminate this?


Some pictures generated with wireshark on Linux client:

Downloading 100MB file from Windows 2008 CIFS file server connected with 1Gbps NIC to Centos 5 Linux client connected with 100Mbps NIC with smbclient:
a graph looking like shark teeth

Downloading 100MB file from Fedora Linux CIFS file server on Samba connected with 1Gbps NIC to Centos 5 Linux client connected with 100Mbps NIC with smbclient (same scale as above):
a graph looking like a seismograph not during earthquake


Here are these programs (linked are compiled using mono's gmcs, require .NET2):

dot_nc.cs

using System;
using System.IO;
using System.Diagnostics;
using System.Net.Sockets;

public class dot_nc
{
 public static void Main(string[] args) {
  string hostname = args[0];
  int port = int.Parse(args[1]);

  Stopwatch stopwatch = new Stopwatch();

  stopwatch.Start();
  TcpClient client = new TcpClient(hostname, port);
  stopwatch.Stop();
  Console.WriteLine("Connection: {0}ms", stopwatch.ElapsedMilliseconds);

  stopwatch.Reset();
  stopwatch.Start();
  byte[] buffer = new byte[4096];
  {
   Stream stdin = Console.OpenStandardInput();
   NetworkStream netout = client.GetStream();
   while ( true ) {
    int bytesread = stdin.Read(buffer, 0, buffer.Length);
    if ( bytesread <= 0 ) {
     break;
    }
    netout.Write(buffer, 0, bytesread);
   }
  }
  stopwatch.Stop();
  Console.WriteLine("Sending: {0}ms", stopwatch.ElapsedMilliseconds);
  client.Close();
 }
}

dot_nc_l.cs

using System;
using System.IO;
using System.Diagnostics;
using System.Net;
using System.Net.Sockets;

public class dot_nc
{
 public static void Main(string[] args) {
  int port = int.Parse(args[0]);

  TcpListener server = new TcpListener(IPAddress.Any, port);
  server.Start();
  TcpClient client = server.AcceptTcpClient();
  NetworkStream netin = client.GetStream();

  byte[] buffer = new byte[4096];

  Stream stdout = Console.OpenStandardOutput();
  int processed_bytes = 0;
  int processed_chunks = 0;
  while ( true ) {
   int bytesread = netin.Read(buffer, 0, buffer.Length);
   if ( bytesread <= 0 ) {
    break;
   }
   stdout.Write(buffer, 0, bytesread);
   processed_bytes += bytesread;
   processed_chunks++;
  }
  netin.Close();
  client.Close();
  server.Stop();
  Console.Error.WriteLine(
   "Received: {0} chunks of data of {1} average size", 
   processed_chunks, processed_bytes/processed_chunks
  );
 }
}

Best Answer

The problem was caused by:

  • too small packet buffers in inexpensive gigabit switch;
  • inadequate congestion avoidance algorithm used in Windows Server 2008 File Services;
  • disabled flow control in network adapter (it was disabled by default).

Because flow control was disabled, Windows was sending packets up to window size in one batch using 1Gbps connection. Since 100Mbps client receive packets much more slowly, almost all data up to window size needed to be buffered by a switch. As this cheap switch has very small buffers (buffer sizes aren't even stated in specifications but it has to be less than 64kB per port, as even disabling window scaling did not help) it had to drop excess packets. Packet loss caused a delay of about 0.25s seen on a graph. But congestion avoidance algorithm, used in File Services, or lack thereof, did not reduce TCP window size, so the next batch of packets wasn't smaller — it congested connection again and again causing congestion collapse.

Standard TCP connections (not File Services) must use different congestion control algorithm and do not get congested repeatably. I suppose treating File Services specially by Windows TCP stack helps in benchmarks against for example Samba.

So the solutions are:

  • Enable flow control in network adapter properties. It isn't an ideal solution, as any File Services transfer to 100Mbps client will also slow down concurrent transfers to 1Gbps clients to less than 100Mbps speeds.

  • Or connect 100Mbps clients to an enterprise class switch with much bigger buffers. This is a solution I've used. I have a 10 year old "3Com SuperStack 3 3300 SM" switch with one 1000Base-SX fiber optic gigabit Ethernet MT-RJ port. I bought a Cisco 1000BASE-SX mini-Gbic module (MGBSX1) with LC port for my Linksys gigabit switch and LC/MT-RJ multi-mode fiber patchcord (about $150 for both) and connected all 100Mbps clients to this 3com switch. I've also enabled flow control but it should not cause slowdowns with no 100Mbps client connected.

Thanks to SpacemanSpiff, whose comments helped to resolve this.

Related Topic