Linux servers seeing bad download performance behind Sonicwall firewall

linuxnetworkingsonicwall

I'm working with a pair of co-located CentOS Linux servers sitting behind a Sonicwall PRO 2040 Enhanced firewall running in transparent bridge mode.

These servers are having a strange problem downloading files more than a few megabytes in size. For example, if I try to wget or FTP a copy of the Linux kernel from kernel.org, the first ~1-2MB will download at 600+K/s, and then throughput will drop off a cliff to 1K/s.

I've reviewed all the firewall configuration settings for anything suspicious, but found nothing. More interestingly, I performed the same download with a Windows server sitting behind the same firewall, and it sailed right through at 600+K/s the whole way.

Has anyone seen this? Where should I start looking to troubleshoot this problem?

Best Answer

We too are experiencing the same problem. Anything larger than what can be transferred in the initial download burst (~3.7mb for us), trickles off to ~1-4kb a second regardless of the bandwidth available.

It seems to be a problem specific to and common with the SonicWall PRO 2040 Firewall - https://discussions.apple.com/message/12250946?messageID=12250946

The root of the problem is the firewall and the best long-term fix is to find a setting on the firewall to allow the TCP Window Scaling option to be turned on and also use the initiating machine's TCP Window Scale Factor correctly in the initialization of the connection.

Though this article refers to routers, the same logic applies to what's happening with the SonicWall Pro 2040 Firewall, http://lwn.net/Articles/92727/:

The details are still being figured out, but it would appear that some routers on the net are rewriting the window scale TCP option on SYN packets as they pass through. In particular, they seem to be setting the scale factor to zero, but leaving the option in place. The receiving side sees the option, and responds with a window scale factor of its own. At this point, the initiating system believes that its scale factor has been accepted, and scales its windows accordingly. The other end, however, believes that the scale factor is zero. The result is a misunderstanding over the real size of the receive window, with the system behind the firewall believing it to be much smaller than it really is. If the expected scale factor (and thus the discrepancy) is large, the result is, at best, very slow communication. In many cases, the small window can cause no packets to be transmitted at all, breaking TCP between the two affected systems entirely.

Similar to what was mentioned above, there are workarounds for individual machines - http://prowiki.isc.upenn.edu/wiki/TCP_tuning_for_broken_firewalls, by turning off the rfc1323 TCP extension, the firewall is never given the opportunity to pass a TCP Window Scale Factor of 0 and instead passes along that the rfc1323 extension is not enabled, presumably using the maximum allowed window size by TCP without the rfc1323 extension, which is 64kb.

Commands we've used on our various machines as a temporary workaround:

Ubuntu 10.10:
Change takes effect immediately:

sudo sysctl -w net.ipv4.tcp_window_scaling=0

Permanent change, after next reboot:

sudo sh -c 'echo "net.ipv4.tcp_window_scaling=0" >> /etc/sysctl.conf'


Mac OSx:
Change takes effect immediately:

sudo sysctl -w net.inet.tcp.rfc1323=0

Permanent change, after next reboot:

sudo sh -c 'echo "net.inet.tcp.rfc1323=0" >> /etc/sysctl.conf'


Win7:
See available options:

netsh interface tcp show global

Disable Command (Persistent):

netsh interface tcp set global autotuning=disabled


In response to why the Windows Server was not having any problems, I found this article - http://msdn.microsoft.com/en-us/library/ms819736.aspx

TCP window scaling is negotiated on demand in Windows Server 2003, based on the value set for the SO_RCVBUF Windows Sockets option when a connection is initiated. Additionally, the Window Scale option is used by default on a connection if the received SYN segment for that connection as initiated by a TCP peer contains the Window Scale option. Windows Server 2003 TCP does not initiate connections with window scaling by default. To instruct the Windows Server 2003 TCP stack to attempt to negotiate a larger receive window size by making use of the Window Scale option, set the Tcp1323Opts registry value to 1.

Related Topic