PFsense https connections unusably slow

Question

I have a very strange issue with PFsense as router running in KVM with CentOS 7. https connections are incredibly slow (10KB/s or less), and uploads over https simply don't work; for example using https://imgur.com over https loads, but uploading an image will take minutes, after which it says it failed.

I have a dual-wan setup with a 192.168.178.x/24 subnet between the PFsense VM and the 2 ADSL router/modems. The router/modem's NAT functionality can not be turned off, so I've simply put them in the same subnet and connected them to eachother with only 1 DHCP server active, the first router sitting on .1 and the second on .2. The PFsense box sits on .5. The private network behind pfsense is 172.16.x.x/16. The PFsense virtual machine runs on a CentOS 7 KVM hypervisor with 2 intel Gbe NICs, bridged using a linux bridge with the VM network cards, using virtIO drivers, if it makes any difference.

I do have a Squidproxy, however it is not enabled for https connections, and https accesses do not appear in Squid's logs, and turning off or removing Squid does not make a difference. Moving myself into the 192.168.178.x/24 subnet before PFsense DOES make a difference however, as suddenly everything runs smoothly again, and any https content loads instantly.

Does anyone have a clue what could be going on? Anything I could try to diagnose? I've tried wireshark and aside for the slowness I don't see anything unusual.. Any suggestions are welcome!

edit: I'm currently running memtest86+ inside a VM (those shouldn't give errors either right?), and I have 1 error so far, although it seems to be outside the range of memory I've granted the VM so I'm a bit confused.. I will update once I have more info. Might run a full memtest on the host later if I can clear users off the host for a moment.

Solution

It's entirely possible that if you're using pfSense 2.2 or later, you're being affected by this. Symptoms would include:

1. Slowness for other VMs hosted on the KVM platform if they need to access a network resource which is on the other side of one of the router interfaces on the pfSense router
2. Physical machines which need to access something across the router are perfectly fast

I am no expert, but my current understanding is that checksums are not correctly calculated for packets that move from one VM to another VM, so either the pfSense router discards them, or the recipient on the other end of the connection discards them, because they believe the packets were mangled in transport (which, I guess, they technically were). There's lots of discussion about it in the thread I linked above, and also in this thread.

To resolve, you'll need to probably disable at least TX checksum offloading on the virtual NICs of the pfSense VM. I'm not sure of the procedure to do that in KVM, since I'm a Xen man, myself. Happy hunting!

Comments

[EEAA]

Is it just https uploads or all uploads?

[Alex]

@EEAA Just https. http uploads work fine, in fact every other protocol I tested works fine except for https, which almost makes me point at squid, except that squid isn't set up to do anything with https at all, and removing it entirely does not change a thing..

[Alex]

It was in load balancing and failover mode (same tier), doesn't change if I change the tier (e.g set one to 1 and the other to 2), nor if I completely disable load balancing. Thanks for thinking along though!

[Alex]

I don't have any kind of traffic shaping enabled, I thought that that could be it for a while too but I couldn't find anything that would cause it... It might be a memory error though, I'm still trying to get a good moment to clear the server out and do a memtest on the hardware itself instead of in a VM. Thanks for thinking along though!

[Alex]

I disabled checksum offloading both both TX/RX and individually, as well as set it to polling. I'm thinking about switching to non-virtIO drivers to see if that solves it, but I'd have to setup a fresh VM first. This is probably related though! Thanks a lot.

[Alex]

I've tested with a fresh config with no ldap/radius installed. Don't think it's an authentication issue.

[Aloha]

Checking "Disable hardware checksum offload" under System/Advanced/Networking made browsing much, MUCH better. (VirtIO, pfSense VM on Proxmox). +1