ethernetprotocol-theory
Ethernet defines a 2 byte header for the payload's length in its frame format.
This header has an ambiguous meaning:
if smaller than or equal to 1500 it represents the payload length.
if greater or equal 1536 (0x600) it is to be considered as an EtherType used for mux/demux of frames to the upper layer.
How come they are in the same header? Are we not interested in expressing the payload's length if, for example, we are using etherType 0x0800 (IPv4)
Identifying the Data Inside an Ethernet Frame
Each data-link header has a field in its header with a code that defines the type of protocol header that follows.
For example, the Destination Service Access Point (DSAP) field has a value of E0, which means that the next header is a Novell IPX header. Why is that? Well, when the IEEE created 802.2, it saw the need for a protocol type field that identified what was inside the field called "data" in an IEEE Ethernet frame.
The IEEE called its Type field the destination service access point (DSAP). When the IEEE first created the 802.2 standard, anyone with a little cash could register favorite protocols with the IEEE and receive a reserved value with which to identify those favorite protocols in the DSAP field. For instance, Novell registered IPX and was assigned hex E0 by the IEEE.
However, the IEEE did not plan for a large number of protocols - and it was wrong. As it turns out, the 1-byte-long DSAP field is not big enough to number all the protocols.
To accommodate more protocols, the IEEE allowed the use of an extra header, called a SubNetwork Access Protocol (SNAP) header. The DSAP field of 0xAA, which implies that a SNAP header follows the 802.2 header, and the SNAP header includes a 2-byte protocol type field. The SNAP protocol type field is used for the same purpose as the DSAP field, but because it is 2 bytes long, all the possible protocols can be identified. For instance, the SNAP type field has a value of 0800, signifying that the next header is an IP header. RFC 1700, "Assigned Numbers" (RFC 1700), lists the SAP and SNAP Type field values and the protocol types that they imply.
See more: SAP Numbers
Responding to individual concerns in the post...
Ideally i would be relying on Path MTU discovery. But since the ethernet packets being generated are too large for any other machine to receive, there is no opportunity for IP Packet too big fragmentation messages to be returned
Based on your diagram, I agree that PMTUD cannot function between two different PCs in the same LAN segment; PCs do not generate ICMP Error messages required by PMTUD.
Some vendors (such as Cisco) have switch models which support ethernet payloads larger than 1500 bytes. Officially IEEE does not endorse this configuration, but the industry has valid needs to judiciously deviate from the original 1500 byte MTU. I have storage LAN / backup networks which leverage jumbo frame for good reason; however, I made sure that all MTUs matched inside the same vlan when I deployed jumbo frames.
The bottom line is that you should never have mismatched ethernet MTUs inside the same ethernet broadcast domain; if you do, it's a bug or configuration error. Regardless of bug or error, you have to solve these problems, sometimes manually.
All that discussion leads to the next question...
Why is there a spec that intentionally creates invalid ethernet frames?
I'm not sure that I agree... I don't see how the IEEE 802.3 series, or RFC 894 create invalid frames. Host implementations or host misconfigurations create invalid frames. To understand whether your implementation is following the spec, we need a lot more evidence...
This diagram is at least prima facie evidence that your MTUs are mismatched inside a broadcast domain...
+------------------+ +----------------+ +------------------+
| Realtek PCIe GBe | | NetGear 10/100 | | Realtek 10/100 |
| (on-board) | | Switch | | (on-board) |
| | +----------------+ | |
| Windows 7 | ^ ^ | |
| | | | | |
| 192.168.1.98/24 |-----------+ +------------| 192.168.1.10/24 |
| MTU = 1504 bytes | | MTU = 1500 bytes |
+------------------+ +------------------+
What was it they [the writers of 'the spec'] expected people to do with devices that generate these too large packets?
MTU 1504 and MTU 1500 within the same broadcast domain is simply a misconfiguration; it should never be expected to work any more than mismatched IP netmasks, or mismatched IP subnets can be expected to work. Your company will have to knuckle-down and fix the root-cause of the MTU mismatches... at this time it's hard to say whether the root cause is user error, an implementation bug, or some combination of the above.
If the affected Windows machines are successfully logging into to an Active Directory Domain, one could write Windows login scripts to automatically fix MTU issues based on some well-constructed tests inside the domain login scripts (assuming the Domain Controller isn't part of the MTU issues).
If the machines are not logging into a domain, manual labor is another option.
Use a layer3 switchNote 1 to build a custom vlan for anything that has broken MTUs and set the layer3 switch's ethernet MTU to match the broken machines; this relies on PMTUD to resolve MTU issues at the IP layer. Layer3 switches generate the ICMP errors required by PMTUD.
This option works best if you can re-address the broken machines with DHCP; and you can identify the broken machines by mac-address.
... why did they bump it up to 1504 bytes, and create invalid packets, in the first place?
Hard to say at this point
How is IEEE 802.1ad (aka VLAN Tagging, QinQ) valid, when the packets are too large?
I haven't seen evidence so far that you're using QinQ; from the limited evidence I have seen so far, you're using simple 802.1q encapsulation, which should work correctly in Windows, assuming the NIC driver supports 802.1q encap.
End Notes:
Note 1Any layer 3 switch should do... Cisco, Juniper, and Brocades all could perform this kind of function.
Best Answer
Ethertype/length actually indicating the frame length is pretty much obsolete, it's redundant. Instead, the frame end is signaled on the physical layer by loss of carrier or a special end-of-data symbol (depending on the specific PHY).
You can read up the formal definition of the Length/Type field in IEEE 802.3 Clause 3.2.6.