pfsense 2.0.2 racoon(ipsec vpn) unreliable

Question

I'm having issues with racoon (ipsec VPN) on pFSense 2.0.2 (and 2.0.1). According to racoon all my tunnels are up (I have about 130 of them), but over time more and more of them won't pass traffic through. If I restart racoon the tunnels start working again, for a period of time.

There is almost no CPU utilization, only about 20% RAM is being used (before or after racoon restart).

On all locations I'm doing DPD, according to PF the tunnels are up.

Just now Nagios was showing I had 54 down locations, restarted racoon, and everything comes back up.

-- Edit -- Also I should note, we currently have pF 1.2.3 running these with absolutely no issues, but I do have the same issue between the two PF boxes (1.2.3 <-> 2.0.2), likely moving to ovpn for this.

-- Edit --

Also noticed today that it only ever drops up to 50-60 of the tunnels over a few hours, and no more.

-- Edit -- From the logs I did find this when pinging a dead location: ERROR: can't start the quick mode, there is no ISAKMP-SA

-- Edit -- What I'm finding is if I login to a device on the remote network and ping the pF network a new Phase2 is created and the tunnel works again. It should be opening the tunnel when I ping the other direction but it's simply not.

-- Edit --

In my case, the modems we are connecting to have a setting for "keep tunnel alive" (not DPD), which seems to work-around this issue pF is having. It seems like pF will not negotiate a phase2 when it's requested, which is extremely curious. I've got Nagios checks happening every couple of minutes trying to go across the tunnel, which should cause pF to do a new P2 (or P1+P2 if required) once the lifetime is expired, but it's just not. According to pF's IPsec status page the tunnel is still alive (probably because the P1 is still valid) when it's quite obviously not.

Solution 1

For anyone looking for resolution try this reference:

• http://forum.pfsense.org/index.php?topic=44895.0

"To resolve this issue disable NAT-T (when pfsense holds the public IP). If that still does not help disable DPD and set 'Negotiation Mode' in Phase 1 to main"

Solution 2

It is probably the security associations going out of sync due little hiccups in the internet connection. Next time it happens, have a look at "status > ipsec > sad". Start pinging the other host, if it is timing out and there are more than 2 (one for each side) then try deleting the dead "data" SAD's, and see if your pings start again. This was very common for me with ipsec.

Also have a look at the pfsense ipsec logs. IPSec logs everything, and if you wish update it here and we can try and help. You can also turn on racoon debugging.

As a long term solution, I use openvpn nowadays, which is built into pfsense, I would recommend setting this up, along side your ipsec and move over to it.

Comments

[cpuguy83]

Unfortunately we are connecting to edge devices (RavenXE cell modems) that only support ipsec or plain GRE tunnels (which I haven't gotten to work). The logs don't show any issues with these locations. Normally if a peer is dead, the next time we try to connect it shows up in the log, creating new SA's and all. With this situation nothing is happening.

[cpuguy83]

It is indeed unchecked.