Routing – Why don’t iperf, scamper and path MTU discovery packet captures agree on the path’s MTU

ipv4routing

Let's do some path MTU discovery between two Debian hosts separated by a Debian router that runs Shorewall-generated iptables rules. Each of the two hosts uses a single Ethernet link while the router uses tagged VLANs over two aggregated Ethernet links.

Using scamper :

root@kitandara:/home/jm# scamper -I "trace -M 10.64.0.2"
traceroute from 10.1.0.5 to 10.64.0.2
 1  10.1.0.1  0.180 ms [mtu: 6128]
 2  10.64.0.2  0.243 ms [mtu: 6128]

Good: 6128 bytes is the expected result (cheap Realtek Ethernet adapters can't handle jumbo frames of a decent size).

Now, let iperf perform a throughput test and tell us about the MTU by the way :

root@kitandara:/home/jm# iperf -c 10.64.0.2 -N -m
------------------------------------------------------------
Client connecting to 10.64.0.2, TCP port 5001
TCP window size: 66.2 KByte (default)
------------------------------------------------------------
[  3] local 10.1.0.5 port 59828 connected with 10.64.0.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1011 MBytes   848 Mbits/sec
[  3] MSS size 6076 bytes (MTU 6116 bytes, unknown interface)

6116 bytes ? Why ?

And now for something completely different, let's see what this session's traffic actually contained :

root@kitandara:/home/jm# tshark -i eth0 -R "(ip.dst == 10.64.0.2) || (ip.src == 10.64.0.2)" | head
Capturing on eth0
  1.308557     10.1.0.5 -> 10.64.0.2    TCP 74 60310 > 5001 [SYN] Seq=0 Win=5340 Len=0 MSS=534 SACK_PERM=1 TSval=101928961 TSecr=0 WS=16
  1.308801    10.64.0.2 -> 10.1.0.5     TCP 74 5001 > 60310 [SYN, ACK] Seq=0 Ack=1 Win=18328 Len=0 MSS=6088 SACK_PERM=1 TSval=3764064056 TSecr=101928961 WS=64

6088 bytes MSS, which means a 6128 MTU… Good. But then why does iperf announce a 6116 bytes MTU ?

At that point thoroughness calls for a closer look at what happens during the scamper trace session :

root@kitandara:/home/jm# tshark -i eth0 -R "(ip.dst == 10.64.0.2) || (ip.src == 10.64.0.2)"
Capturing on eth0
  0.000000     10.1.0.5 -> 10.64.0.2    UDP 58 Source port: 43870  Destination port: 33435
  0.000175     10.1.0.1 -> 10.1.0.5     ICMP 86 Time-to-live exceeded (Time to live exceeded in transit)
  0.050358     10.1.0.5 -> 10.64.0.2    UDP 58 Source port: 43870  Destination port: 33436
  0.050592    10.64.0.2 -> 10.1.0.5     ICMP 86 Destination unreachable (Port unreachable)
  0.099790     10.1.0.5 -> 10.64.0.2    UDP 6142 Source port: 43870  Destination port: 33437
  0.100912    10.64.0.2 -> 10.1.0.5     ICMP 590 Destination unreachable (Port unreachable)

All those packets have a udp.length of 24 except the two last which have a udp.length of 6108… But then how does scamper tell us that the path MTU is 6128 ?

6108, 6116, 6128… So many MTU to choose from !

Best Answer

Very interesting.

MSS (maximum segment size) = MTU - IP header = 6076.

6076 + 40 = 6116.

Could it be Debian is using the IP options fields in the IP header? That might be the extra 12 bytes...

Related Solutions

Routing – Path MTU discovery and retransmission

If a PC receives an ICMP Fragmentation required message, the PC must resend any traffic. Normally only TCP handles retransmission (including loss from MTU issues); TCP also includes support for Path MTU discovery.

In case of non-TCP applications which are written to support PMTU discovery and retransmission, the PC could know which packets to resend because ICMP errors include the context of the packet which generated the error... this is a dump from tshark -Vi eth0 icmp... Look below MTU of next hop, it shows that this ICMP message was received in response to a UDP packet to 172.16.255.1 (with DF set), which was too large.

Internet Protocol, Src: 172.16.1.1 (172.16.1.1), Dst: 172.16.1.5 (172.16.1.5)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0xc0 (DSCP 0x30: Class Selector 6; ECN: 0x00)
        1100 00.. = Differentiated Services Codepoint: Class Selector 6 (0x30)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 56
    Identification: 0x6297 (25239)
    Flags: 0x00
        0.. = Reserved bit: Not Set
        .0. = Don't fragment: Not Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 255
    Protocol: ICMP (0x01)
    Header checksum: 0xfe46 [correct]
        [Good: True]
        [Bad : False]
    Source: 172.16.1.1 (172.16.1.1)
    Destination: 172.16.1.5 (172.16.1.5)
Internet Control Message Protocol
    Type: 3 (Destination unreachable)
    Code: 4 (Fragmentation needed)
    Checksum: 0x8813 [correct]
    MTU of next hop: 1420
    Internet Protocol, Src: 172.16.1.5 (172.16.1.5), Dst: 172.16.255.1 (172.16.255.1)
        Version: 4
        Header length: 20 bytes
        Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
            0000 00.. = Differentiated Services Codepoint: Default (0x00)
            .... ..0. = ECN-Capable Transport (ECT): 0
            .... ...0 = ECN-CE: 0
        Total Length: 1500
        Identification: 0x3aff (15103)
        Flags: 0x02 (Don't Fragment)
            0.. = Reserved bit: Not Set
            .1. = Don't fragment: Set
            ..0 = More fragments: Not Set
        Fragment offset: 0
        Time to live: 1
            [Expert Info (Note/Sequence): "Time To Live" only 1]
                [Message: "Time To Live" only 1]
                [Severity level: Note]
                [Group: Sequence]
        Protocol: UDP (0x11)
        Header checksum: 0xe0ea [correct]
            [Good: True]
            [Bad : False]
        Source: 172.16.1.5 (172.16.1.5)
        Destination: 172.16.255.1 (172.16.255.1)
    User Datagram Protocol, Src Port: 60742 (60742), Dst Port: 33437 (33437)
        Source port: 60742 (60742)
        Destination port: 33437 (33437)
        Length: 1480
        Checksum: 0xf73f [unchecked, not all data available]
            [Good Checksum: False]
            [Bad Checksum: False]

Exactly when is PMTUD performed? (Path MTU discovery)

The answer is simple: whenever the host pleases. Really. It's that simple.

The explanation below assumes an IPv4-only environment, since IPv6 does away with fragmentation in the routers (forcing the host to always deal with fragmentation and MTU discovery).

There is no strict rule that governs when (or even if) a host does Path MTU Discovery. The reason that PMTUD surfaced is that fragmentation is considered harmful for various reasons. To avoid packet fragmentation, the concept of PMTUD was brought to life as a workaround. Of course, a nice operating system should use PMTUD to minimize fragmentation.

So, naturally, the exact semantics of when PMTUD is used depend on the sender's operating system - in particular, the socket implementation. I can only speak for the specific case of Linux, but other UNIX variants are probably not very different.

In Linux, PMTUD is controlled by the IP_MTU_DISCOVER socket option. You can retrieve its current status with getsockopt(2) by specifying the level IPPROTO_IP and the IP_MTU_DISCOVER option. This option is valid for SOCK_STREAM sockets only (a SOCK_STREAM socket is a two-way, connection-oriented, reliable socket; in practice it's a TCP socket, although other protocols are possible), and when set, Linux will perform PMTUD exactly as defined in RFC 1191.

Note that in practice, PMTUD is a continuous process; packets are sent with the DF bit set - including the 3-way handshake packets - you can think of it as a connection property (although an implementation may be willing to accept a certain degree of fragmentation at some point and stop sending packets with the DF bit set). Thus, PMTUD is just a consequence of the fact that everything on that connection is being sent with DF.

What if you don't set IP_MTU_DISCOVER?

There's a default value. By default, IP_MTU_DISCOVER is enabled on SOCK_STREAM sockets. This can be read or changed by reading /proc/sys/net/ipv4/ip_no_pmtu_disc. A zero value means that IP_MTU_DISCOVER is enabled by default in new sockets; a non-zero means the opposite.

What about connectionless sockets?

This is tricky because connectionless, unreliable sockets do not retransmit lost segments. It becomes the user's responsibility to packetize the data in MTU-sized chunks. Also, the user is expected to make the necessary retransmits in case of a Message too big error. So, essentially user code must reimplement PMTUD. Nevertheless, if you're up for the challenge, you can force the DF bit by passing the IP_PMTUDISC_DO flag to setsockopt(2).

The bottomline

The host decides when (and if) to use PMTUD
When it uses PMTUD, it's like a connection attribute, it happens continuously (but at any point the implementation is free to stop doing so)
Different operating systems use different approaches, but usually, reliable, connection-oriented sockets perform PMTUD by default, whereas unreliable, connectionless sockets don't

Best Answer

Related Solutions

Routing – Path MTU discovery and retransmission

Exactly when is PMTUD performed? (Path MTU discovery)

Related Topic