Let's say i configure the MSS > IP MTU. Even if it doesn't make sense, would this be possible?
Assuming i then try to send a file of the size of MSS or bigger, the network layer will receive the data (plus TCP headers) from the transport layer and append it's own headers. The MTU of the datagram will now be greater than the configured MTU. Will the local network layer fragment the datagram or discard it (sending some kind of error to the upper layer)?
Best Answer
It depends which MTU you are actually talking about:
If you are talking about the layer-2 MTU, the IP packets are fragmented:
Let's say you want to send an UDP packet with 5000 bytes length over Ethernet. In this case one IP packet is generated that is 5068 bytes long. This packet is then split into fragments of 1500 bytes length. The fragments are transmitted.
If you are talking about the maximum IP packet length:
If one of the hosts involved does not support IP packets of the given length, the data transfer will fail. If the destination host does not support IP packets of 5068 bytes length (in the example), the packet cannot be received.
I read about an ISP using DS-Lite which does not support DS-Lite tunneling (IPv4-in-IPv6) if the resulting packet is longer than 1500 bytes. As a result a TCP/IPv4 packet with more than 1460 bytes length cannot be routed.
I think that most operating systems will silently limit the MSS to the maximum size of IP packets supported on the own host. However, many TCP/IP implementations do not send an MSS option in the first packet so it is possible that a TCP packet is longer than the maximum IP packet size allowed by the receiver. (Exactly this seem to be the case with the ISP mentioned above.)