Assuming you mean to protect confidentiality of the communication at IP layer with IPsec:
How would the underlying network be able to differentiate between UDP
and TCP since they're at the transport layer.
The next header field of the ESP header tells you the type of payload.
If you use tunnel mode (which is custom for VPNs), then without the necessary keys you cannot decide what's at transport layer because the next header field will tell you just that there's a whole IP packet encapsulated.
If you use transport mode, then the next header field tells you the type of payload at transport layer.
Will we still have TCP and UDP when we move the IPv6(Although I see
that IPsec has been made optional for IPv6)?
TCP and UDP are agnostic to the layer-3 protocol. In fact, TCP and UDP (and SCTP and DCCP) exist also for IPv6.
What seems to puzzle you is that in IPsec tunnel (VPN) mode there is no way to inspect the content. This is supposed to happen at the tunnel end-points. An organization that is worried by this loss of control should not allow IPsec that is not under it's own control.
Further reading: An Illustrated Guide to IPsec
Suppose I'm receiving IPv4 UDP packets whose payload is less than 18 octets, so when they are transmitted over Ethernet they have some trailing padding. The equipment generating said packets includes the length of the padding in the IP header's total length field. So, for example, a packet with a 7-octet-long payload will not have an IP total length of 35, but rather 46.
Where did you get that idea?
Ethernet is a layer-2 protocol, and it doesn't know about the layer-3 protocol. Ethernet can carry any number of layer-3 protocols (IPv4, IPX, IPv6, AppleTalk, etc.), and it knows nothing about the layer-3 protocols or headers for the layer-3 protocols, so it has no way to change a field in a layer-3 header.
Conversely, the layer-3 protocol has no idea which layer-2 protocol (ethernet, Wi-Fi, token ring, frame relay, ATM, PPP, etc.) carries its packets.
The ethernet padding is for ethernet frames at layer-2, not the layer-3 packets.
Edit:
You completely changed the meaning of the question, which is very bad form, especially when you already got an answer to the original question. You should start a new question for a different question, not change the original question.
The device in the middle that changes the IPv4 header total length field must change the IPv4 header checksum, and the UDP checksum (if it used, but it is optional for IPv4, and it is often not used) is not computed using the IPv4 total length field or checksum, so it would not change.
If the IPv4 total length field is changed, then IPv4 will send its original payload (the UDP datagram) and the padding to UDP.
The UDP header has its own length field. If the device does not modify this field, then the UDP will be correct and UDP will send the correct number of octets (not including the padding) to the application, but if the device changes the UDP length field, then the UDP checksum will be recomputed and UDP will send the original UDP payload, plus the padding to the application, possibly causing a problem for the application.
Best Answer
That's just the way it is designed. You can use a checksum but you don't need to.
The application might not care so much about data corruption or your application-layer protocol might include a much better integrity check than UDP's very simple checksum. Also, there may be scenarios where it is preferrable to receive anything rather than nothing at all.
Using no checksum in the UDP header can also be a benefit - since datagrams failing checksum verification are simply dropped, the receiving application doesn't get to see them.
By explicitly not using the checksum option, the sender application can have damaged datagrams delivered to the receiver application just the same. Now, if the sender application also includes forward error correction (FEC) information in the UDP payload (on the application layer), that allows the receiver application to repair damaged data and receive correct data after all.