C++ – Increasing TCP Window Size

csocketstcp

I have some doubts over increasing TCP Window Size in application. In my C++ software application, we are sending data packets of size around 1k from client to server using TCP/IP blocking socket. Recently I came across this concept TCP Window Size. So I tried increasing the value to 64K using setsockopt() for both SO_SNDBUF and SO_RCVBUF. After increasing this value, I get some improvements in performance for WAN connection but not in LAN connection.

As per my understanding in TCP Window Size,

Client will send data packets to server. Upon reaching this TCP Window Size, it will wait to make sure ACK received from the server for the first packet in the window size. In case of WAN connection, ACK is getting delayed from the server to the client because of latency in RTT of around 100ms. So in this case, increasing TCP Window Size compensates ACK wait time and thereby improving performance.

I want to understand how the performance improves in my application.

In my application, even though TCP Window Size (Both Send and Receive Buffer) is increased using setsockopt at socket level, we still maintain the same packet size of 1k (i.e the bytes we send from client to server in a single socket send). Also we disabled Nagle algorithm (inbuilt option to consolidate small packets into a large packet thereby avoiding frequent socket call).

My doubts are as follows:

Since I am using blocking socket, for each data packet send of 1k, it should block if ACK doesn't come from the server. Then how does the performance improve after improving the TCP window Size in WAN connection alone ? If I misunderstood the concept of TCP Window Size, please correct me.
For sending 64K of data, I believe I still need to call socket send function 64 times ( since i am sending 1k per send through blocking socket) even though I increased my TCP Window Size to 64K. Please confirm this.
What is the maximum limit of TCP window size with windows scaling enabled with RFC 1323 algorithm ?

I am not so good in my English. If you couldn't understand any of the above, please let me know.

Best Answer

First of all, there is a big misconception evident from your question: that the TCP window size is what is controlled by SO_SNDBUF and SO_RCVBUF. This is not true.

What is the TCP window size?

In a nutshell, the TCP window size determines how much follow-up data (packets) your network stack is willing to put on the wire before receiving acknowledgement for the earliest packet that has not been acknowledged yet.

The TCP stack has to live with and account for the fact that once a packet has been determined to be lost or mangled during transmission, every packet sent, from that one onwards, has to be re-sent since packets may only be acknowledged in order by the receiver. Therefore, allowing too many unacknowledged packets to exist at the same time consumes the connection's bandwidth speculatively: there is no guarantee that the bandwidth used will actually produce anything useful.

On the other hand, not allowing multiple unacknowledged packets at the same time would simply kill the bandwidth of connections that have a high bandwidth-delay product. Therefore, the TCP stack has to strike a balance between using up bandwidth for no benefit and not driving the pipe aggressively enough (and thus allowing some of its capacity to go unused).

The TCP window size determines where this balance is struck.

What do `SO_SNDBUF` and `SO_RCVBUF` do?

They control the amount of buffer space that the network stack has reserved for servicing your socket. These buffers serve to accumulate outgoing data that the stack has not yet been able to put on the wire and data that has been received from the wire but not yet read by your application respectively.

If one of these buffers is full you won't be able to send or receive more data until some space is freed. Note that these buffers only affect how the network stack handles data on the "near" side of the network interface (before they have been sent or after they have arrived), while the TCP window affects how the stack manages data on the "far" side of the interface (i.e. on the wire).

Answers to your questions

No. If that were the case then you would incur a roundtrip delay for each packet sent, which would totally destroy the bandwidth of connections with high latency.
Yes, but that has nothing to do with either the TCP window size or with the size of the buffers allocated to that socket.
According to all sources I have been able to find (example), scaling allows the window to reach a maximum size of 1GB.

Related Solutions

Macos – Who is listening on a given TCP port on Mac OS X

On macOS Big Sur and later, use this command:

sudo lsof -i -P | grep LISTEN

or to just see just IPv4:

sudo lsof -nP -i4TCP:$PORT | grep LISTEN

On older versions, use one of the following forms:

sudo lsof -nP -iTCP:$PORT | grep LISTEN
sudo lsof -nP -i:$PORT | grep LISTEN

Substitute $PORT with the port number or a comma-separated list of port numbers.

Prepend sudo (followed by a space) if you need information on ports below #1024.

The -n flag is for displaying IP addresses instead of host names. This makes the command execute much faster, because DNS lookups to get the host names can be slow (several seconds or a minute for many hosts).

The -P flag is for displaying raw port numbers instead of resolved names like http, ftp or more esoteric service names like dpserve, socalia.

See the comments for more options.

For completeness, because frequently used together:

To kill the PID:

sudo kill -9 <PID>
# kill -9 60401

Python: Binding Socket: “Address already in use”

Try using the SO_REUSEADDR socket option before binding the socket.

comSocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

Edit: I see you're still having trouble with this. There is a case where SO_REUSEADDR won't work. If you try to bind a socket and reconnect to the same destination (with SO_REUSEADDR enabled), then TIME_WAIT will still be in effect. It will however allow you to connect to a different host:port.

A couple of solutions come to mind. You can either continue retrying until you can gain a connection again. Or if the client initiates the closing of the socket (not the server), then it should magically work.