Electrical – PCIe Channel Multiplexing

pcie

According to Wikipedia

PCI Express 3.0's 8 GT/s bit rate effectively delivers 985 MB/s per
lane

985MBps * 8 = 7880MBps (63Gbps).

If I am using a 10Gbps Ethernet NIC with PCIe 3.0 8x interface and I am sending or receiving data at full 10Gbps line rate, I assume that my data between the OS and network card is being multiplexed over multiple PCIe channels in the motherboard slot my card is in.

Assuming that is correct, how does PCIe maintain packet ordering when multiplexing over multiple channels or is there no data ordering support? How am I getting 10Gbps of throughput with 8Gbps per channel?

Best Answer

The card is effectively a Serializer/Deserializer (SerDes for short). One possible implementation is there's a clock for the parallel bus running at PCIe 3.0 speed, which latches the incoming 8 bits of data. Then internally there is a shift register running at a faster clock (10Gb ethernet speeds) which just shifts that out. The process can be performed in reverse to convert ethernet data into PCIe signals.

There is a strong correlation between which PCIe pin corresponds to which bit of the parallel buffer, which is what guarantees the correct ordering in both transmit and receive mode (ex.: pin 0 is always bit 0, pin 1 is always bit 1, etc.). Parallelism is exposed at the bit level.

There is more complications when dealing with the actual PCIe/10G ethernet protocols, (10Gb ethernet uses a 64b/66b encoding), but this basic picture mostly sums up the broad idea.

Related Topic