Good day all! I will try and make this short and sweet. My current setup is:
XS708T NETGEAR 10gbE switch
ESXi Machine #1:
- 2x Intel 10gbE
- pfSense (VM) with WAN on one vSwitch, LAN on other vSwitch, NIC's above respectively for each.
- pfSense, Ubuntu Server (x2), FreeNAS server, Windows Server VM's
ESXi Machine #2:
- 1x Intel 10gb (SFP+)
- Ubuntu Server (x2), FreeNAS server, Windows Server VM's
VLAN's: set up on pfSense and switch
- 10 = VM's
- 20 = LAN_OTHER
- 30 = WIFI
- 99 = ADMIN
So here is the issue: using iperf between both Ubuntu VM's on separate AND same ESXi machines are able to reach the full 10GB/s throughput when on the same VLAN. I have ran in both directions as well. Watching systat -ifstat on pfSense, I can verify no traffic is traveling through pfSense. If I try and go between two Ubuntu VM's on the SAME or DIFFERENT ESXi machines, but with different VLAN's (ie: 10 & 99), It drops down to about 2 – 3 GB/s throughput. During this test, I can see all the traffic traveling through pfSense.
So my thought process is this; my switch has been verified capable of handing the 10GB/s throughput when on the same VLAN. I have monitored pfSense during all the test to verify that I'm not hitting any sort of hardware (CPU) limits.
So is this an issue with my setup, or possibly my pfSense VM? Any and all help is appreciated! Thank you!
Best Answer
The pfSense VM has quite a bit of overhead to fight with - ingress packets go through the host NIC, host vSwitch, are copied to the VM, pass the guest stack, are processed and egress packets take the same way back. All this happens in software and requires quite a few context switches. These cause latency on the packet and - through the resulting bandwidth-delay product - may limit the overall throughput for a delay-sensitive transport protocol such as TCP.
There are three basic approaches: reduce latency (reserve host resources for VM, use a paravirtualized vNIC or a dedicated NIC with passthrough, fine-tune NUMA setup), increase the TCP receive window with TCP window scaling, or use multiple TCP flows at the same time.