Implementing jumbo frames on one interface and not the other

First, it might be best to explain what jumbo frame ethernet is. Ethernet is a layer 2 networking technology and its Protocol Data Unit (PDU) is a frame. For reference, an L3PDU (IP layer) is a packet, and an L4PDU (tcp/udp) is a segment.

An ethernet frame (there are several types of ethernet but we can generalize here) consists of a header (containing, among other things, a source MAC, a destination MAC, an 802.1q VLAN tag, etc) the data, or paylod, of the frame, and a CRC checksum used to validate the successful transmission of the frame.

The original ethernet specified a frame size (the valume of data in the entire frame, including the header and checksum) as 1500 bytes (or possibly 1518, have to look it up). This number struck a balance between amount of data to send at once and the likelihood of that transmission failing or colliding and having to be retransmitted. With the advent of fast, full duplex LANs, people realized that performance could be improved by increasing the ethernet frame size. The traditional size of jumbo frames are 9000 bytes per frame, though this is mostly convention.

On a rock solid, full duplex LAN (or VLAN) in which all elements are expecting to receive jumbo frame ethernet, it actually does improve performance. The problem with this scenario is if you introduce a network element or end device which doesn't expect it. In the best case, it will result in a performance degradation as packets are lost because receiving devices are expecting only 1518 bytes in a frame.

Now to your specific questions:

What would be the best way to implement Jumbo Frames on a network?

This is a subjective question. At my place of business we chose to implement it only where we knew we had all the variable under control and we knew it would help. To do this we implemented it in a special "private" vlan that only specific devices could access via their second NICs. Specifically, we put the second NIC of our file servers and application servers into this new VLAN and then changed all references to the IP scheme used in this VLAN. That allows us to narrowly target (no ones going to plug a desktop machine into this VLAN) the specific area we know would most benefit (the highest utilization data links in our infrastructure). This maximizes gain while minimizing risk.

More specifically, on the network side (using IOS), we built VLANs dedicated to the jumbo frame devices, then added "mtu 9000" to their vlan definition. Every interface on the switch that would be using this network was put in this vlan using something like "switchport access vlan 11". On the linux machines (which have eth0 connected to the standard network and eth1 connected to the jumbo frame network) we added "MTU=9000" to /etc/sysconfig/network-scripts/ifcfg-eth1. Because we never route these packets (it's impossible for anything not directly connected to the jumbo frame VLAN to speak with a NIC on the jumbo frame VLAN) we never had to worry about a router configuration.

From what I can tell in order for it to work properly all network gear on the network must support Jumbo Frames. Is this true?

Yeah, pretty much. All network "clients" (by which I means servers/desktops/IPKVMs/IP environmental monitors, etc) must speak it also or, as mentioned above, you'll have lots of semi-reachable machines (they'll ping, and any L3 or L4PDU that's less than 1500 bytes will succeed, which means, as an example, your mail server will ping, and you'll be able to hand deliver what will likely be a small test message. But when you try to deliver a real mail (the one with the excel attachment that is pushed the frame size >1500 bytes) it will mysteriously fail).

If I have specific gear (e.g. network printer) that cannot be updated to GB ethernet will this prevent me from enabling Jumbo Frames?

If that's the case, here's what I would do (assuming network gear that can handle this):

• build two VLANs, one with jumbo frame and one without
• assign all your network devices to one vlan or the other
• in your router, and switches, implement the jumbo frame vlan and change the frame size on any network clients.

This means that you will no longer have a flat L2 topology on your network. For instance, if from your jumbo-frame enabled server you want to print to your non-jumbo frame printer, the packets will have to be routed (travel through your router, the frames rewritten into a more conventional size, and then sent to the printer on the other VLAN). This means that communication between your jumbo frame and non-jumbo frame machines will be slightly poorer than it was before, but the data transfer rates between all devices on the jumbro frame VLAN will be better. It's really just a judgement call.

What are some of the gotcha's of enabling Jumbo Frames?

Hopefully covered above. Good luck!

Jumbo frames may make some difference but your throughput problems indicate a much more severe issue. You can enable Jumbo frames in ESXi but it requires the use of the vCLI command line tools - you can find specific instructions in this VMware ESXi config document.

There are some possible causes.

You might have your data going in and out of the ESXi host - in that case Converter would be copying the data from within the VM in the ESXi host back to the Management Interface via your physical network. Given that it's a 100Megabit uplink I'd still expect you to get a couple of Megabytes/sec rather than the 4Megabit/sec you report.

Your ESX host NICs might not actually be negotiating the 100Mbps/Full duplex settings correctly with the switch - make sure both the switch and the pNIC settings on your ESXi host are set correctly.

Converter isn't terribly efficient in terms of throughput but if you are using block based disk copying (rather than file level) it is OK (transfer rate will be >50% of the link bandwidth's maximum - say 4 Meg/sec on a 100Mbps network, 40Meg/sec on GigE). If your copy is using file level copying then things will be a lot slower.

All of this activity is putting a fair amount of additional load on the disk subsystem your VM's are stored on. If you are running all of this off fairly slow storage (say a handful of SATA drives in RAID 5) then it's possible that the disk is thrashing but for a healthy storage setup this sort of thing shouldn't be a stress.

I think the problem is with your virtual networking though - assuming that it is you should consider the following:

If your ESXi Management Port is on the same virtual switch as your VM's production network port group then the traffic should loop back internally within the Virtual Switch. If that's not happening then I'd start checking whether VLAN's are configured on the ports\port groups or check whether your ip-addressing is causing the traffic to think it has to exit the switch before coming back in (e.g. if you have the Management port on a different subnet to the VM Network and are relying on an external router to allow them to communicate). If you suspect that your network isn't doing the above correctly then you can put the Source and target VM's on the same subnet as the management port and connect them to a VM Port Group on the same vSwitch as the Management port then you should get traffic between the various systems (the source, the converter VM and the ESX host) to remain within the confines of the vSwitch. Move the VM port groups rather than messing with the Management port - if you make a mistake with that you'll have to go back to the ESXi's physical console to fix things and it's best to avoid taking any risks with that.

Also shut down as much as you can before you start just in case something like a Backup process is hogging all the Management port network bandwidth etc.