mtuvxlan
I have a cluster of servers interconnected with 10 Gbps fabric interfaces (ens1f0, MTU 1500). I've set up a VXLAN interface across the cluster using default settings (MTU 1450). For reasons of compatibility, I want to have MTU 1500 with my VXLAN interface. Is it possible?
I'm currently using this command on all servers to create the VXLAN interface:
ip link add vxlan0 type vxlan id 10 group 239.1.1.1 dstport 0 dev ens1f0
There is a very important distinction between the Internet, and your own LAN, (or even WAN circuits.)
On the Internet, 1500 is the max MTU. (End of discussion.)
On your own LAN, (or in some cases WAN too) you can do whatever you want. As technologies expanded, 1500 MTU was no longer needed. So things like jumbo frames came into use.
I had a similar issue using a Static VTI IPSec tunnel. Initially the tunnel interface IP MTU was set to 1400 bytes with the "Crypto IPSec DF-bit clear" command set under the global configuration. The expected behaviour was for packets received (being forwarded to the VTI) should have the DF bit cleared and allow the packet to be fragmented before encryption is required. We found that any packets arriving with DF bit set that exceeded the tunnel IP MTU were being dropped and not switched to the tunnel interface.
After a long discussion with Cisco TAC it was explained that that VTI tunnels do not support fragmentation before encryption (as per bug ID CSCsr97396/CSCsh30577), This is apparently undocumented in any publically available Cisco documents including configuration guides... poor show Cisco considering this bug has been known since 2007!!!
The fix is to ensure the tunnel interface IP MTU is large enough (changed IP MTU to 1500) not to cause fragmentation. In the solution we deployed the egress interface is a point to point ATM interface (OC12/STM4 ATM SPA) with a default MTU of 4470 so no post encryption fragmentation will occur.
BTW, we are using ASR1006 router with ESP-10, RP2 with IOS-XE 3.13 if that helps.
Best Answer
A frame's or packet's payload cannot exceed the maximum frame/packet size minus the overhead.
VXLAN is VLAN-enabled L2-tunneling over UDP. It encapsulates Ethernet frames in UDP datagrams, in turn encapsulated in IP, in turn encapsulated in Ethernet (simplest case). Since that overhead eats away 50 bytes of the payload capacity, you'd need to use "baby giants", slightly oversized outer frames on the outside, with an MTU capability of 1550 bytes. That way, an inner frame (minus overhead) could still support a payload of 1500 bytes.
Check your equipment and the entire(!) path between VXLAN endpoints if they support those baby giants and how to configure them. Note that with any non-standard frame size, all nodes in each intermediate segment need to be able to handle them. There is no negotiation for frame sizes, and nodes generally drop oversized frames as giants.