You're right, if you don't take any measures this could happen. It's a violation of the acceptable use policy of most IXP's I know, but still you want to prevent it from happening.
Your first solution is a good thing to do and will solve your problem. Just make sure you don't keep track of session state in iptables, that will probably kill performance or even your router.
You could consider to do outbound filtering as well in a similar way: do not allow packets to leave your network originating from unknown sources. This will prevent prevent hosts in your network from sending spoofed IP packets, which are commonly used in DDoS attacks.
I wouldn't implement the second solution. It's complicated and doesn't scale well if you have multiple routers handling your transits and peerings or if you have a large number of peering sessions (a couple of hunderds on an IXP isn't that uncommon).
On all the hardware router platforms I know this problem is solved in configuration by configuring RPF on the outbound interface and/or by writing filters.
You have hit what I have always considered a pain point in the naming here. Let me start with a small correction to what you said: technically speaking, the MTU is the maximum size of an L3PDU inside an Ethernet frame. The L3PDU is made up of IP headers and payload. So payload =/= MTU.
Now, when you talk about this informally, you often just say "payload", without clarifying whether you are talking about the payload of the Ethernet frame, or the effective payload of the L3PDU inside it. And this brings me to the issue: the confusion comes from the fact that some of the names actually overlap. When you talk about TCP MTU, you are effectively talking about the maximum packet size, which is a value that is independent from the physical Ethernet MTU, which remains 1500 bytes.
In summary, the real MTU is the physical one, i.e. the maximum size of the IP packet inside an Ethernet frame. When you talk about MTU with regards to other network layers, you are effectively talking about the maximum packet size, which doesn't necessarily have the same value of the physical MTU.
This article talks about this in a bit more detail, but here is an excerpt that is relevant to this question:
Higher-level network protocols like TCP/IP can be configured with a maximum packet size, a parameter independent of the physical layer MTU over which TCP/IP runs. Unfortunately, many network devices use the terms interchangeably. On both home broadband routers and Xbox Live enabled game consoles, for example, the parameter called MTU is in fact the maximum TCP packet size and not the physical MTU.
Best Answer
Pete said:
Short answer
Routing protocols are some of the most fundamental building blocks on the internet; we need them to be very reliable in every possible case. It does no good to bring up an OSPF or EIGRP adjacency on a mismatched MTU.
Routing protocols must remove any potential mismatched MTUs from the router's forwarding path.
Long answer
I can think of three possible situations where you'd find mismatched IGP MTUs...
IP MTUs are directly correlated to Layer2 MTUs (at least for Case 1, above). No matter what we do, we are always at the mercy of mitigating the problems from unintentional Layer2 MTU mismatches, since there is no Layer2 MTU discovery mechanism (unlike IP, which has ICMP error messages).
This means that we have to do everything possible to avoid Layer2 MTU mismatches, even if Cases 2 and 3 above are casualties of mitigating problems with Case number 1. Case 1 has colossal implications unless we solve it; i.e. black-holing all traffic just because we permitted mismatched MTUs.
We're always limited to the least common denominator on the link. Frames larger than the receive MTU of an interface are silently discarded, and the router has no way of knowing whether the MTU was intentionally mismatched, or whether it happened accidentally.
Consequently, EIGRP and OSPF require valid Layer2 adjacencies Note 1 (including MTUs).
Quoting John Moy (OSPF's author) in RFC 2329 Page 4:
Also quoting him from the OSPF mailing list:
Note 1 some people misunderstand the meaning of adjacency as strictly an IP routing protocol concept. This assertion misses the reality that everything (including IP) requires matching layer2 MTUs, for Layer2 domains to work properly.
One of the most important functions of a routing protocol is building a valid FIB / CEF / forwarding table. That table maps the information learned via routing protocols to layer2 rewrite info. Those Layer2 relationships on the same physical link are what Cisco also calls adjacencies.