Ethernet – Jumbo Ethernet Frame Deployment

Terms

Physical interface -- a layer-2 port or switchport. Refers to switch uplinks and downlinks in the context of this document. May or may not be performing 802.1q tagging.

Logical interface -- a layer-3 port or routed port. Refers to server interfaces, router interfaces, switch virtual interfaces and subinterfaces. Anything with an IP address.

Standard domain -- a set of layer-3-adjacent VLANs which operate at the standard Ethernet payload of 1500 bytes.

Jumbo domain -- a set of layer-3-adjacent VLANs which operate at an Ethernet payload MTU greater than 1500 bytes, usually 9000 bytes.

Network Topology

Jumbo frames are not a mix-and-match technology. Jumbo frames are deployed to a clearly delineated segment of the network which is bounded by a router. This layer-3 boundary is essential to support interoperability with the standard domain. The router provides fragmentation and ICMP responses for Path MTU Discovery that are not provided at the lower levels of the networking stack. All communication within the jumbo domain will enjoy support for larger payload, while all communication between standard and jumbo domains will occur at the standard payload size. Communication between discontiguous jumbo domains also occurs at the standard payload size.

Jumbo VLANs

Jumbo frames are not a mix-and-match technology. This point is worth repeating. A given VLAN belongs to the standard domain or the jumbo domain, never both. This means that all physical interfaces that carry a jumbo VLAN must permit jumbo frames. If a jumbo frame encounters a physical interface that does not permit jumbo frames it will be dropped. Likewise, all logical interfaces on a jumbo VLAN must be configured with the exact same MTU value.

Physical Interfaces

Physical interfaces can carry any number of standard and jumbo VLANs as long as jumbo frames are permitted on the interface. Also, both types of VLANs may be trunked to a host or router. It is only important that the logical interfaces on the host or router be configured with the proper MTU value for each respective VLAN. In this sense, there exists a disconnect between physical interfaces and logical interfaces. A physical interface must meet or exceed the jumbo MTU, it does not need to be an exact match.

Logical Interfaces

Jumbo frames are not a mix-and-match technology. There it is again. For all intents and purposes every physical interface in the network can permit jumbo frames whether it carries a standard VLAN, a jumbo VLAN or both. The logical interfaces is where the exact MTU match really matters. This is also where it can get a bit confusing. For instance, configuring a Cisco 6500 SVI for MTU 9000 permits an Ethernet payload of 9000 bytes while configuring an HP NIC team for MTU 9014 permits an Ethernet payload of 9000 bytes. This is because the Cisco value only specifies the payload and the HP value specifies the payload plus Ethernet header. It is important to be aware of these details to get the MTU match exactly right.

Validation

It is useful to be aware of methods for validating that a jumbo VLAN has been properly deployed. Ping tests are sufficient, but one must take care to set the DF bit. Both Cisco and Windows ping utilities offer a method to do this, but once again the method for counting the bytes differs. The length parameter on Cisco includes the ICMP payload, ICMP header and IP header. The length parameter on Windows only specifies the ICMP payload. A length of 9000 on Cisco and a length of 8972 on Windows both produce an Ethernet frame of length 9018 with a payload of 9000 bytes.

Conclusion

Hopefully someone out there will find this diatribe useful. In networking, I think we all know the importance of doing it right the first time. Please feel free to leave comments regarding the accuracy of information, potential improvements and additional references.

References

http://en.wikipedia.org/wiki/Jumbo_frame