Terms
Physical interface -- a layer-2 port or switchport. Refers to switch uplinks and downlinks in the context of this document. May or may not be performing 802.1q tagging.
Logical interface -- a layer-3 port or routed port. Refers to server interfaces, router interfaces, switch virtual interfaces and subinterfaces. Anything with an IP address.
Standard domain -- a set of layer-3-adjacent VLANs which operate at the standard Ethernet payload of 1500 bytes.
Jumbo domain -- a set of layer-3-adjacent VLANs which operate at an Ethernet payload MTU greater than 1500 bytes, usually 9000 bytes.
Network Topology
Jumbo frames are not a mix-and-match technology. Jumbo frames are deployed to a clearly delineated segment of the network which is bounded by a router. This layer-3 boundary is essential to support interoperability with the standard domain. The router provides fragmentation and ICMP responses for Path MTU Discovery that are not provided at the lower levels of the networking stack. All communication within the jumbo domain will enjoy support for larger payload, while all communication between standard and jumbo domains will occur at the standard payload size. Communication between discontiguous jumbo domains also occurs at the standard payload size.
Jumbo VLANs
Jumbo frames are not a mix-and-match technology. This point is worth repeating. A given VLAN belongs to the standard domain or the jumbo domain, never both. This means that all physical interfaces that carry a jumbo VLAN must permit jumbo frames. If a jumbo frame encounters a physical interface that does not permit jumbo frames it will be dropped. Likewise, all logical interfaces on a jumbo VLAN must be configured with the exact same MTU value.
Physical Interfaces
Physical interfaces can carry any number of standard and jumbo VLANs as long as jumbo frames are permitted on the interface. Also, both types of VLANs may be trunked to a host or router. It is only important that the logical interfaces on the host or router be configured with the proper MTU value for each respective VLAN. In this sense, there exists a disconnect between physical interfaces and logical interfaces. A physical interface must meet or exceed the jumbo MTU, it does not need to be an exact match.
Logical Interfaces
Jumbo frames are not a mix-and-match technology. There it is again. For all intents and purposes every physical interface in the network can permit jumbo frames whether it carries a standard VLAN, a jumbo VLAN or both. The logical interfaces is where the exact MTU match really matters. This is also where it can get a bit confusing. For instance, configuring a Cisco 6500 SVI for MTU 9000 permits an Ethernet payload of 9000 bytes while configuring an HP NIC team for MTU 9014 permits an Ethernet payload of 9000 bytes. This is because the Cisco value only specifies the payload and the HP value specifies the payload plus Ethernet header. It is important to be aware of these details to get the MTU match exactly right.
Validation
It is useful to be aware of methods for validating that a jumbo VLAN has been properly deployed. Ping tests are sufficient, but one must take care to set the DF bit. Both Cisco and Windows ping utilities offer a method to do this, but once again the method for counting the bytes differs. The length parameter on Cisco includes the ICMP payload, ICMP header and IP header. The length parameter on Windows only specifies the ICMP payload. A length of 9000 on Cisco and a length of 8972 on Windows both produce an Ethernet frame of length 9018 with a payload of 9000 bytes.
Conclusion
Hopefully someone out there will find this diatribe useful. In networking, I think we all know the importance of doing it right the first time. Please feel free to leave comments regarding the accuracy of information, potential improvements and additional references.
References
http://en.wikipedia.org/wiki/Jumbo_frame
Regardless of which Ethernet PHY you use, the nominal speed is always the effective bandwidth at the top of the physical layer. Whether or not FEC is used, which PCS code etc doesn't matter. The symbol rate on the physical wire/fiber is appropriately higher to accomodate.
So, a maximum-sized Ethernet frame over 25GE takes exactly (1538 * 8 / 25,000,000,000) ≈ 49.2 μs. (1500 bytes payload, 18 bytes L2 overhead, 20 bytes L1 overhead including IPG), regardless of the PHY and its options.
For instance, the maximum effective throughput for TCP over IPv4 over 25GE is (1460 / 1538) * 25,000,000,000 / 8 ≈ 2.967 GB/s. (The window scaling option is almost always required.)
As JFL has pointed out, the real-world throughput depends on your hardware. Not all 25GE hardware is non-blocking. For instance, a 25G NIC requires at least PCIe 3.0 x4, PCIe 2.0 x8 or equivalent slot (and appropriate backend) for non-blocking transfer.
RS-FEC is applied between the PCS (66b/64b encoding) and the PMA sublayers (check IEEE 802.3 Clause 108). It transcodes 64b/66b to 256b/257b (108.5.2.3) to allow headroom for RS, so it doesn't require a higher baudrate.
Re follow-on: PCS line code, RS-FEC transcoding etc are applied in the physical layer. They don't change the data link layer frame in any way. Generally, PCS, RS-FEC etc only matter within the lower PHY. They have absolutely no effect on the upper PHY and above.
Best Answer
Sequential delivery of frames is considered a hard invariant. This means that in any properly working network, the frames should always be delivered in the order they are received.
When you get above L2, this is not always the case as packets can be received out of order. However if you do tunnel L2 traffic over L3, then the process ideally should account for making sure that sequential delivery of the frames is enforced even if the packets are received out of order. In practice this is often not the case.