L2TP/IPSec stopped working after openssl upgrade

Question

VPN connections from my MacBook / iOS devices to a Debian server having openswan / xl2tp were working just fine until I used apt-get to upgrade everything due to openssl heartbleed announcement.

Now the VPN connection stopped working with the following in the server auth.log:

Apr 11 10:32:50 linode pluto[7868]: packet from x.x.x.x:500: received Vendor ID payload [RFC 3947] method set to=109
Apr 11 10:32:50 linode pluto[7868]: packet from x.x.x.x:500: received Vendor ID payload [draft-ietf-ipsec-nat-t-ike] method set to=110
Apr 11 10:32:50 linode pluto[7868]: packet from x.x.x.x:500: ignoring unknown Vendor ID payload [8f8d83826d246b6fc7a8a6a428c11de8]
Apr 11 10:32:50 linode pluto[7868]: packet from x.x.x.x:500: ignoring unknown Vendor ID payload [439b59f8ba676c4c7737ae22eab8f582]
Apr 11 10:32:50 linode pluto[7868]: packet from x.x.x.x:500: ignoring unknown Vendor ID payload [4d1e0e136deafa34c4f3ea9f02ec7285]
Apr 11 10:32:50 linode pluto[7868]: packet from x.x.x.x:500: ignoring unknown Vendor ID payload [80d0bb3def54565ee84645d4c85ce3ee]
Apr 11 10:32:50 linode pluto[7868]: packet from x.x.x.x:500: ignoring unknown Vendor ID payload [9909b64eed937c6573de52ace952fa6b]
Apr 11 10:32:50 linode pluto[7868]: packet from x.x.x.x:500: received Vendor ID payload [draft-ietf-ipsec-nat-t-ike-03] meth=108, but already using method 110
Apr 11 10:32:50 linode pluto[7868]: packet from x.x.x.x:500: received Vendor ID payload [draft-ietf-ipsec-nat-t-ike-02] meth=107, but already using method 110
Apr 11 10:32:50 linode pluto[7868]: packet from x.x.x.x:500: received Vendor ID payload [draft-ietf-ipsec-nat-t-ike-02_n] meth=106, but already using method 110
Apr 11 10:32:50 linode pluto[7868]: packet from x.x.x.x:500: ignoring Vendor ID payload [FRAGMENTATION 80000000]
Apr 11 10:32:50 linode pluto[7868]: packet from x.x.x.x:500: received Vendor ID payload [Dead Peer Detection]
Apr 11 10:32:50 linode pluto[7868]: "L2TP-PSK-NAT"[1] x.x.x.x #1: responding to Main Mode from unknown peer x.x.x.x
Apr 11 10:32:50 linode pluto[7868]: "L2TP-PSK-NAT"[1] x.x.x.x #1: transition from state STATE_MAIN_R0 to state STATE_MAIN_R1
Apr 11 10:32:50 linode pluto[7868]: "L2TP-PSK-NAT"[1] x.x.x.x #1: STATE_MAIN_R1: sent MR1, expecting MI2
Apr 11 10:32:50 linode pluto[7868]: "L2TP-PSK-NAT"[1] x.x.x.x #1: message ignored because it contains an unknown or unexpected payload type (ISAKMP_NEXT_SAK) at the outermost level
Apr 11 10:32:50 linode pluto[7868]: "L2TP-PSK-NAT"[1] x.x.x.x #1: sending notification INVALID_PAYLOAD_TYPE to x.x.x.x:500
Apr 11 10:32:53 linode pluto[7868]: "L2TP-PSK-NAT"[1] x.x.x.x #1: message ignored because it contains an unknown or unexpected payload type (ISAKMP_NEXT_SAK) at the outermost level
Apr 11 10:32:53 linode pluto[7868]: "L2TP-PSK-NAT"[1] x.x.x.x #1: sending notification INVALID_PAYLOAD_TYPE to x.x.x.x:500
Apr 11 10:32:56 linode pluto[7868]: "L2TP-PSK-NAT"[1] x.x.x.x #1: message ignored because it contains an unknown or unexpected payload type (ISAKMP_NEXT_SAK) at the outermost level

and then the connection breaks.

Same type of connection used to generate this in the logs only 10 days ago:

Apr  1 04:16:04 linode pluto[3093]: "L2TP-PSK-NAT"[37] y.y.y.y #43: responding to Main Mode from unknown peer y.y.y.y
Apr  1 04:16:04 linode pluto[3093]: "L2TP-PSK-NAT"[37] y.y.y.y #43: transition from state STATE_MAIN_R0 to state STATE_MAIN_R1
Apr  1 04:16:04 linode pluto[3093]: "L2TP-PSK-NAT"[37] y.y.y.y #43: STATE_MAIN_R1: sent MR1, expecting MI2
Apr  1 04:16:04 linode pluto[3093]: "L2TP-PSK-NAT"[37] y.y.y.y #43: NAT-Traversal: Result using draft-ietf-ipsec-nat-t-ike (MacOS X): peer is NATed
Apr  1 04:16:04 linode pluto[3093]: "L2TP-PSK-NAT"[37] y.y.y.y #43: transition from state STATE_MAIN_R1 to state STATE_MAIN_R2
Apr  1 04:16:04 linode pluto[3093]: "L2TP-PSK-NAT"[37] y.y.y.y #43: STATE_MAIN_R2: sent MR2, expecting MI3
Apr  1 04:16:04 linode pluto[3093]: "L2TP-PSK-NAT"[37] y.y.y.y #43: ignoring informational payload, type IPSEC_INITIAL_CONTACT msgid=00000000
Apr  1 04:16:04 linode pluto[3093]: "L2TP-PSK-NAT"[37] y.y.y.y #43: Main mode peer ID is ID_IPV4_ADDR: '192.168.2.101'
Apr  1 04:16:04 linode pluto[3093]: "L2TP-PSK-NAT"[37] y.y.y.y #43: switched from "L2TP-PSK-NAT" to "L2TP-PSK-NAT"
Apr  1 04:16:04 linode pluto[3093]: "L2TP-PSK-NAT"[38] y.y.y.y #43: deleting connection "L2TP-PSK-NAT" instance with peer y.y.y.y {isakmp=#0/ipsec=#0}
Apr  1 04:16:04 linode pluto[3093]: "L2TP-PSK-NAT"[38] y.y.y.y #43: transition from state STATE_MAIN_R2 to state STATE_MAIN_R3

etc.

My questions are:

1. What's the most apparent meaning of INVALID_PAYLOAD_TYPE error?
2. Other than downgrading openswan, what are my best options to investigate and fix this issue?

Solution

I just encounterd the same problem today and I seems its caused by the latest security update of debian wheezy for openswan. When you do a dpkg -l | grep openswan I assume you have 1:2.6.37-3+deb7u1 installed.

To get it working with your iPad/ IPhone again you have to downgrade openswan on your server with apt-get install openswan=1:2.6.37-3.

Of course this is only an ugly workaround and I'm not sure if this is bug in the latest openswan or IOS, but let's hope either will fix it asap.

Comments

[Frank Thomas]

it looks like it can;t negotiate an IKE method (Internetwork Key Exchange), and despite trying multiple methods, still can't agree on one between the two peers.

[Dennis Kreminsky]

I assume that IKE's must be configurable, somewhere?

[Frank Thomas]

nope, looks like an auto-negotiation fallback mechanism. probably defined by international standards. I'm thinking its because you have a mismatch between the software component capabilities on the peers. make sure both sides of the connection have been upgraded.

[Dennis Kreminsky]

client side is mac os x / ios, so not much I can do really, what puzzles me is that it was working just fine, and googling only gives raspberry users that have similar issues and just downgrade to get it fixed.

[Halfgaar]

I installed the patched deb from here and it works: bugs.debian.org/cgi-bin/bugreport.cgi?bug=744717

[KernelSanders]

This fixed the issue for me as well. For anyone looking for openswan 1:2.6.37-3 for the raspberry pi, it is available here: snapshot.raspbian.org/201403301125/raspbian/pool/main/o/…

[gucki]

Better yet, switch to strongswan if possible. Openswan is going to be removed from next debian release, afaik.