Sockets – TCP loopback connection vs Unix Domain Socket performance

Turns out socat can be used to achieve this:

socat TCP-LISTEN:1234,reuseaddr,fork UNIX-CLIENT:/tmp/foo

And with a bit of added security:

socat TCP-LISTEN:1234,bind=127.0.0.1,reuseaddr,fork,su=nobody,range=127.0.0.0/8 UNIX-CLIENT:/tmp/foo

These examples have been tested and work as expected.

1) In what cases, and why, would communicating over loopback be slower than over the network?

Loopback puts the packet setup+tcp chksum calculation for both tx+rx on the same machine, so it needs to do 2x as much processing, while with 2 machines you split the tx/rx between them. This can have negative impact on loopback.

2) When sending as fast as possible, why does toggling TCP_NODELAY have so much more of an impact on maximum throughput over loopback than over the network?

Not sure how you've come to this conclusion, but the loopback vs network are implemented very differently, and if you try to push them to the limit, you will hit different issues. Loopback interfaces (as mentioned in answer to 1) cause tx+rx processing overhead on the same machine. On the other hand, NICs have a # of limits in terms of how many outstanding packets they can have in their circular buffers etc which will cause completely different bottlenecks (and this varies greatly from chip to chip too, and even from the switch that's between them)

3) How can we detect and analyze TCP congestion control as a potential explanation for the poor performance?

Congestion control only kicks in if there is packet loss. Are you seeing packet loss? Otherwise, you're probably hitting limits on the tcp window size vs network latency factors.

4) Does anyone have any other theories as to the reason for this phenomenon? If yes, any method to prove the theory?

I don't understand the phenomenon you refer to here. All I see in your table is that you have some sockets with a large send buffer - this can be perfectly legitimate. On a fast machine, your application will certainly be capable of generating more data than the network can pump out, so I'm not sure what you're classifying as a problem here.

One final note: small messages create a much bigger performance hit on your network for various reasons, such as:

• there is a fixed per packet overhead (for mac+ip+tcp headers), and the smaller the payload is, the more overhead you're going to have.
• many NIC limitations are relative to the # of outstanding packets, which means you'll hit NIC bottlenecks with much less data when using smaller packets.
• the network itself as per-packet overhead, so the max amount of data you can pump through the network is dependent on the size of the packets again.