PFsense https connections unusably slow

httpskvm-virtualizationnetworkingpfsense

I have a very strange issue with PFsense as router running in KVM with CentOS 7. https connections are incredibly slow (10KB/s or less), and uploads over https simply don't work; for example using https://imgur.com over https loads, but uploading an image will take minutes, after which it says it failed.

I have a dual-wan setup with a 192.168.178.x/24 subnet between the PFsense VM and the 2 ADSL router/modems. The router/modem's NAT functionality can not be turned off, so I've simply put them in the same subnet and connected them to eachother with only 1 DHCP server active, the first router sitting on .1 and the second on .2. The PFsense box sits on .5. The private network behind pfsense is 172.16.x.x/16.
The PFsense virtual machine runs on a CentOS 7 KVM hypervisor with 2 intel Gbe NICs, bridged using a linux bridge with the VM network cards, using virtIO drivers, if it makes any difference.

I do have a Squidproxy, however it is not enabled for https connections, and https accesses do not appear in Squid's logs, and turning off or removing Squid does not make a difference. Moving myself into the 192.168.178.x/24 subnet before PFsense DOES make a difference however, as suddenly everything runs smoothly again, and any https content loads instantly.

Does anyone have a clue what could be going on? Anything I could try to diagnose? I've tried wireshark and aside for the slowness I don't see anything unusual.. Any suggestions are welcome!

edit:
I'm currently running memtest86+ inside a VM (those shouldn't give errors either right?), and I have 1 error so far, although it seems to be outside the range of memory I've granted the VM so I'm a bit confused.. I will update once I have more info. Might run a full memtest on the host later if I can clear users off the host for a moment.

Best Answer

It's entirely possible that if you're using pfSense 2.2 or later, you're being affected by this. Symptoms would include:

  1. Slowness for other VMs hosted on the KVM platform if they need to access a network resource which is on the other side of one of the router interfaces on the pfSense router
  2. Physical machines which need to access something across the router are perfectly fast

I am no expert, but my current understanding is that checksums are not correctly calculated for packets that move from one VM to another VM, so either the pfSense router discards them, or the recipient on the other end of the connection discards them, because they believe the packets were mangled in transport (which, I guess, they technically were). There's lots of discussion about it in the thread I linked above, and also in this thread.

To resolve, you'll need to probably disable at least TX checksum offloading on the virtual NICs of the pfSense VM. I'm not sure of the procedure to do that in KVM, since I'm a Xen man, myself. Happy hunting!