FreeBSD – Why Change net.inet.tcp.tcbhashsize for Performance Tuning

freebsdperformance-tuningsysctl

In virtually every FreeBSD network tuning document I can find:

# /boot/loader.conf
net.inet.tcp.tcbhashsize=4096

This is usually paired with some unhelpful statement like "TCP control-block hash table tuning" or "Set this to a reasonable value." man 4 tcp isn't much help either:

tcbhashsize         Size of the TCP control-block hash table (read-only).
                    This may be tuned using the kernel option TCBHASHSIZE
                    or by setting net.inet.tcp.tcbhashsize in the
                    loader(8).

The only document I can find that touches on this mysterious thing is the Protocol Control Block Lookup subsection beneath Transport Layer in Optimizing the FreeBSD IP and TCP Stack, but its description is more about potential bottlenecks in using it. It seems tied to matching new TCP segments to their listening sockets, but I'm not sure how.

What exactly is the TCP Control Block used for? Why would you want to set its hash size to 4096 or any other particular number?

Best Answer

It's more like computer science question. Especially if you want to dig into hash tables and big-O notations.

The answer is:
If you are handling many TCP sessions on sever you really want to look up connection's tcp parameters in O(1) time instead of O(n). FreeBSD uses chaining to resolve hash table collisions. So if there is lots of connection there will be lots of collisions and so instead of O(1) hash table lookup you'll need to do an linear chain lookup with O(n) complexity.

Parameter you mentioned - tcbhashsize is basically number of buckets in hash table.
On our servers it's set to pretty high values like 16384 and even higher. With that setting we are handling about 60,000 connections per server.

Each entry in hash table by itself currently on x86_64 uses 252 bytes (tcp_inpcb) + 688 bytes(tcpcb) of kernel memory for each entry (kmem size is 512G in amd64 since 7.2+ IIRC). It can be viewed via vmstat -z.

About structure of TCP Control block you can read FreeBSD sources: tcp_var.h or read TCP/IP Illustrated, Volume 2: The Implementation By Gary R. Wright, W. Richard Stevens

Related Topic