Ethernet CRC in TCPDump – Understanding the Details

ethernet

I am testing the ethernet transmission on my Digilent Nexus4 DDR FPGA. I created a really simple packet with a broadcast destination address and some made up ethernet type. Here is the packet as captured in tcpdump.

05:58:22.148546 00:00:00:00:00:00 (oui Ethernet) > Broadcast, ethertype Unknown (0xebeb), length 66: 
    0x0000:  ffff ffff ffff 0000 0000 0000 ebeb 0000
    0x0010:  0000 dead beef 0000 0000 0000 0000 0000
    0x0020:  0000 0000 0000 0000 0000 0000 0000 0000
    0x0030:  0000 0000 0000 0000 0001 0203 0405 0607
    0x0040:  27ae

In the packet I am sending the last few bytes are 0x01 0x02 0x03 0x04 0x05 0x06 0x07 which I can clearly see in the packet. However instead of ending at 0x07 the captured packet includes 0x27ae which happens to be the first two bytes of my CRC code. When I check the output signals using chipscope it looks like I am just transmitting 0x07 and then another 4 bytes for the crc as expected. I have the ethernet from my fpga plugged into a usb ethernet adapter on my desktop since the ethernet port on my motherboard is used to connect to the internet through my router. The usb nic card is the "AmazonBasics USB 3.0 to 10/100/1000 Gigabit Ethernet Adapter".

I then thought that maybe the phy on the card was creating a crc for me so I purposefully flipped the bits in the first byte of the crc and I stopped getting packets. In ifconfig I could see errors packets being dropped. So it looks like that although the ethernet adapter on my pc is checking the full 4 byte crc it is also transmitting the first half of it to the computer.

The next experiment I tried was instead of connecting my FPGA to the usb ethernet adapter I plugged it directly into my router. Since I am using a broadcast ethernet destination I figured the packet would get routed back to my pc through the nic built into its motherboard thereby skipping the usb ethernet adapter. This worked and the last byte in the packet was in fact 0x07, and I did not see any of my crc in the dump. Then while keeping the fpga connected to the router I disconnected the motherboard from the router and connected the usb ethernet adapter to the router. Now when I received packets from the fpga -> router -> usb ethernet it included part of the crc again. So it looks like the problem lies with my usb adapter and not the FPGA.

I thought that maybe my usb adapter was having issues, but I can browse the internet with no issues over it. However, most protocols I can think of like TCP or UDP have a length field so they are probably ignoring the extra two bytes of data anyway. Is this a normal behavior for some nic cards? I am thinking it might have something to do with latency where the card just transmits part of the crc before realizing that it is in the crc.

Best Answer

This is likely a driver issue with the NIC in capture or promiscuous mode. It's also possible that it happens in normal mode as well since - as you've pointed out - most protocols keep track of their data themselves and trailed junk is just ignored.

The NIC has to recognize the FCS field in a reliable fashion, otherwise it'd drop all frames. On reception, FCS is calculated 'on the fly' over all bytes of the frame. When the end-of-frame is detected the FCS value has to the the CRC residue or "magic number" 0xC704DD7B. If correct the NIC then strips the last four bytes and passes the frame to the OS. Apparently, this NIC doesn't strip all FCS bytes.

Regarding Path MTU Discovery

Ideally i would be relying on Path MTU discovery. But since the ethernet packets being generated are too large for any other machine to receive, there is no opportunity for IP Packet too big fragmentation messages to be returned

Based on your diagram, I agree that PMTUD cannot function between two different PCs in the same LAN segment; PCs do not generate ICMP Error messages required by PMTUD.

Jumbo frames

Some vendors (such as Cisco) have switch models which support ethernet payloads larger than 1500 bytes. Officially IEEE does not endorse this configuration, but the industry has valid needs to judiciously deviate from the original 1500 byte MTU. I have storage LAN / backup networks which leverage jumbo frame for good reason; however, I made sure that all MTUs matched inside the same vlan when I deployed jumbo frames.

Mismatched MTUs within a broadcast domain

The bottom line is that you should never have mismatched ethernet MTUs inside the same ethernet broadcast domain; if you do, it's a bug or configuration error. Regardless of bug or error, you have to solve these problems, sometimes manually.

All that discussion leads to the next question...

Why is there a spec that intentionally creates invalid ethernet frames?

I'm not sure that I agree... I don't see how the IEEE 802.3 series, or RFC 894 create invalid frames. Host implementations or host misconfigurations create invalid frames. To understand whether your implementation is following the spec, we need a lot more evidence...

This diagram is at least prima facie evidence that your MTUs are mismatched inside a broadcast domain...

+------------------+      +----------------+     +------------------+
| Realtek PCIe GBe |      | NetGear 10/100 |     | Realtek 10/100   |
|       (on-board) |      |     Switch     |     |     (on-board)   |
|                  |      +----------------+     |                  |
| Windows 7        |           ^    ^            |                  |
|                  |           |    |            |                  |
| 192.168.1.98/24  |-----------+    +------------| 192.168.1.10/24  |
| MTU = 1504 bytes |                             | MTU = 1500 bytes |
+------------------+                             +------------------+

How should an 802.3-compliant implementation respond to MTU mismatches?

What was it they [the writers of 'the spec'] expected people to do with devices that generate these too large packets?

MTU 1504 and MTU 1500 within the same broadcast domain is simply a misconfiguration; it should never be expected to work any more than mismatched IP netmasks, or mismatched IP subnets can be expected to work. Your company will have to knuckle-down and fix the root-cause of the MTU mismatches... at this time it's hard to say whether the root cause is user error, an implementation bug, or some combination of the above.

If the affected Windows machines are successfully logging into to an Active Directory Domain, one could write Windows login scripts to automatically fix MTU issues based on some well-constructed tests inside the domain login scripts (assuming the Domain Controller isn't part of the MTU issues).

If the machines are not logging into a domain, manual labor is another option.

Other possibilities to contain the damage

Use a layer3 switch^{Note 1} to build a custom vlan for anything that has broken MTUs and set the layer3 switch's ethernet MTU to match the broken machines; this relies on PMTUD to resolve MTU issues at the IP layer. Layer3 switches generate the ICMP errors required by PMTUD.

This option works best if you can re-address the broken machines with DHCP; and you can identify the broken machines by mac-address.

... why did they bump it up to 1504 bytes, and create invalid packets, in the first place?

Hard to say at this point

802.1ad vs 802.1q

How is IEEE 802.1ad (aka VLAN Tagging, QinQ) valid, when the packets are too large?

I haven't seen evidence so far that you're using QinQ; from the limited evidence I have seen so far, you're using simple 802.1q encapsulation, which should work correctly in Windows, assuming the NIC driver supports 802.1q encap.

End Notes:

^{Note 1}_{Any layer 3 switch should do... Cisco, Juniper, and Brocades all could perform this kind of function.}

Switch – Difference between CRC errors and FCS errors

It was my understanding that CRC is the algorithm that FCS uses to do its check. Meaning that the Frame Relay, PPP and other protocols have the CRC to be used as the algorithm to calculate the FCS as defined in their standard. Other protocols could include a FCS but use a different algorithm besides CRC.