Linux – Full MTU not utilized with jumbo frames enabled

First, it might be best to explain what jumbo frame ethernet is. Ethernet is a layer 2 networking technology and its Protocol Data Unit (PDU) is a frame. For reference, an L3PDU (IP layer) is a packet, and an L4PDU (tcp/udp) is a segment.

An ethernet frame (there are several types of ethernet but we can generalize here) consists of a header (containing, among other things, a source MAC, a destination MAC, an 802.1q VLAN tag, etc) the data, or paylod, of the frame, and a CRC checksum used to validate the successful transmission of the frame.

The original ethernet specified a frame size (the valume of data in the entire frame, including the header and checksum) as 1500 bytes (or possibly 1518, have to look it up). This number struck a balance between amount of data to send at once and the likelihood of that transmission failing or colliding and having to be retransmitted. With the advent of fast, full duplex LANs, people realized that performance could be improved by increasing the ethernet frame size. The traditional size of jumbo frames are 9000 bytes per frame, though this is mostly convention.

On a rock solid, full duplex LAN (or VLAN) in which all elements are expecting to receive jumbo frame ethernet, it actually does improve performance. The problem with this scenario is if you introduce a network element or end device which doesn't expect it. In the best case, it will result in a performance degradation as packets are lost because receiving devices are expecting only 1518 bytes in a frame.

Now to your specific questions:

What would be the best way to implement Jumbo Frames on a network?

This is a subjective question. At my place of business we chose to implement it only where we knew we had all the variable under control and we knew it would help. To do this we implemented it in a special "private" vlan that only specific devices could access via their second NICs. Specifically, we put the second NIC of our file servers and application servers into this new VLAN and then changed all references to the IP scheme used in this VLAN. That allows us to narrowly target (no ones going to plug a desktop machine into this VLAN) the specific area we know would most benefit (the highest utilization data links in our infrastructure). This maximizes gain while minimizing risk.

More specifically, on the network side (using IOS), we built VLANs dedicated to the jumbo frame devices, then added "mtu 9000" to their vlan definition. Every interface on the switch that would be using this network was put in this vlan using something like "switchport access vlan 11". On the linux machines (which have eth0 connected to the standard network and eth1 connected to the jumbo frame network) we added "MTU=9000" to /etc/sysconfig/network-scripts/ifcfg-eth1. Because we never route these packets (it's impossible for anything not directly connected to the jumbo frame VLAN to speak with a NIC on the jumbo frame VLAN) we never had to worry about a router configuration.

From what I can tell in order for it to work properly all network gear on the network must support Jumbo Frames. Is this true?

Yeah, pretty much. All network "clients" (by which I means servers/desktops/IPKVMs/IP environmental monitors, etc) must speak it also or, as mentioned above, you'll have lots of semi-reachable machines (they'll ping, and any L3 or L4PDU that's less than 1500 bytes will succeed, which means, as an example, your mail server will ping, and you'll be able to hand deliver what will likely be a small test message. But when you try to deliver a real mail (the one with the excel attachment that is pushed the frame size >1500 bytes) it will mysteriously fail).

If I have specific gear (e.g. network printer) that cannot be updated to GB ethernet will this prevent me from enabling Jumbo Frames?

If that's the case, here's what I would do (assuming network gear that can handle this):

• build two VLANs, one with jumbo frame and one without
• assign all your network devices to one vlan or the other
• in your router, and switches, implement the jumbo frame vlan and change the frame size on any network clients.

This means that you will no longer have a flat L2 topology on your network. For instance, if from your jumbo-frame enabled server you want to print to your non-jumbo frame printer, the packets will have to be routed (travel through your router, the frames rewritten into a more conventional size, and then sent to the printer on the other VLAN). This means that communication between your jumbo frame and non-jumbo frame machines will be slightly poorer than it was before, but the data transfer rates between all devices on the jumbro frame VLAN will be better. It's really just a judgement call.

What are some of the gotcha's of enabling Jumbo Frames?

Hopefully covered above. Good luck!

The key here is that you can control your little segment of the network and enable large MTUs but you can't control the path your packets take over the internet and certainly can't control the configuration of the routers your packets will pass through. Most internet routers aren't configured above 1500 so you aren't going to have much luck with this solution. Worse, sometimes larger packets will actually get dropped by routers that don't support jumbo frames so I think you'd actually find things are worse if you try sending jumbo frames to the internet.

Jumbo frames are great on your internal network - especially for networks doing streaming or iSCSI.