IPSec tunnel fails in phase 2

Question

We are trying to establish a tunnel between our EC2 Instance and remote Cisco 3000 series device where it is failing for Phase2. Below is the scenario:

FTP Server(ec2-ubuntu) <---->VPN Server(ec2-ubuntu) <------> Cisco 3000 <---> Client Servers (E-IP) (E-IP) (Peer IP) (Public IPs)

Requirement : 1. Client Servers should reach FTP server via Elastic IP over IPSEC Tunnel. 2. IKE and ESP Parameters looks fine based on details provided by client.

================IPSEC Configuration START=========
config setup
 nat_traversal=yes
 protostack=netkey
 plutostderrlog=/var/log/pluto.log
 nhelpers=0

 conn example-one
  authby=secret
  auto=start
  type=tunnel
  left=%defaultroute
  leftid=107.23.xx.xx
  leftsourceip=107.23.xx.xx
  leftsubnet=107.23.xxx.xxx/32
  right=144.230.xx.xx
  rightid=144.230.xx.xx
  rightsourceip=144.230.xx.xx
  rightsubnets={144.226.xxx.xx/32 144.226.xxx.xx/32}
  keyexchange=ike
  ike=aes256-sha1;modp1024
  phase2=esp
  phase2alg=aes256-sha1;modp1024
  aggrmode=no
  pfs=no

=============END=================

==========iptables nat rules on VPN Server ======

iptables -t nat -A PREROUTING -d 107.23.xxx.xxx -j DNAT --to-destination 10.0.10.20 
iptables -t nat -A POSTROUTING -d 10.0.10.20 -j SNAT --to-source 107.23.xxx.xxx

10.0.10.20 <<------ Private IP of FTP Server

107.23.xxx.xxx <<------- EIP of FTP Server

Belos is the ipsec status on my vpn-server.

000 Total IPsec connections: loaded 1, active 1
000  
000 State Information: DDoS cookies not required, Accepting new IKE connections
000 IKE SAs: total(1), half-open(0), open(0), authenticated(1), anonymous(0)
000 IPsec SAs: total(1), authenticated(1), anonymous(0)
000  
000 #2: "example-one":4500 STATE_QUICK_I2 (sent QI2, IPsec SA established); EVENT_SA_REPLACE in 28045s; newest IPSEC; eroute owner; isakmp#1; idle; import:admin initiate
000 #2: "example-one" [email protected] [email protected] [email protected] [email protected] ref=0 refhim=4294901761 Traffic: ESPout=0B ESPin=0B! ESPmax=4194303B 
000 #1: "example-one":4500 STATE_MAIN_I4 (ISAKMP SA established); EVENT_SA_REPLACE in 2604s; newest ISAKMP; lastdpd=-1s(seq in:0 out:0); idle; import:admin initiate
000  
000 Bare Shunt list:
000

Below are pluto logs.

Apr  3 12:44:28: adding interface lo/lo ::1:500
Apr  3 12:44:28: | setup callback for interface lo:500 fd 22
Apr  3 12:44:28: | setup callback for interface lo:4500 fd 21
Apr  3 12:44:28: | setup callback for interface lo:500 fd 20
Apr  3 12:44:28: | setup callback for interface eth0:4500 fd 19
Apr  3 12:44:28: | setup callback for interface eth0:500 fd 18
Apr  3 12:44:28: | setup callback for interface eth0:4500 fd 17
Apr  3 12:44:28: | setup callback for interface eth0:500 fd 16
Apr  3 12:44:28: loading secrets from "/etc/ipsec.secrets"
Apr  3 12:44:28: loading secrets from "/etc/ipsec.d/example.secrets"
Apr  3 12:44:28: "example-one" #1: initiating Main Mode
Apr  3 12:44:28: "example-one" #1: received Vendor ID payload [RFC 3947]
Apr  3 12:44:28: "example-one" #1: received Vendor ID payload [FRAGMENTATION c0000000]
Apr  3 12:44:28: "example-one" #1: enabling possible NAT-traversal with method RFC 3947 (NAT-Traversal)
Apr  3 12:44:28: "example-one" #1: transition from state STATE_MAIN_I1 to state STATE_MAIN_I2
Apr  3 12:44:28: "example-one" #1: STATE_MAIN_I2: sent MI2, expecting MR2
Apr  3 12:44:28: "example-one" #1: received Vendor ID payload [Cisco-Unity]
Apr  3 12:44:28: "example-one" #1: received Vendor ID payload [XAUTH]
Apr  3 12:44:28: "example-one" #1: ignoring unknown Vendor ID payload [5397e372bf085cf3a0b093e1623498c2]
Apr  3 12:44:28: "example-one" #1: ignoring Vendor ID payload [Cisco VPN 3000 Series]
Apr  3 12:44:28: "example-one" #1: NAT-Traversal: Result using RFC 3947 (NAT-Traversal) sender port 500: I am behind NAT
Apr  3 12:44:28: "example-one" #1: transition from state STATE_MAIN_I2 to state STATE_MAIN_I3
Apr  3 12:44:28: "example-one" #1: STATE_MAIN_I3: sent MI3, expecting MR3
Apr  3 12:44:28: "example-one" #1: received Vendor ID payload [Dead Peer Detection]
Apr  3 12:44:28: | protocol/port in Phase 1 ID Payload is 17/0. accepted with port_floating NAT-T
Apr  3 12:44:28: "example-one" #1: Main mode peer ID is ID_IPV4_ADDR: '144.230.xxx.xxx'
Apr  3 12:44:28: "example-one" #1: transition from state STATE_MAIN_I3 to state STATE_MAIN_I4
Apr  3 12:44:28: "example-one" #1: STATE_MAIN_I4: ISAKMP SA established {auth=PRESHARED_KEY cipher=aes_256 integ=sha group=MODP1024}
Apr  3 12:44:28: "example-one" #2: initiating Quick Mode PSK+ENCRYPT+TUNNEL+UP+IKEV1_ALLOW+IKEV2_ALLOW+SAREF_TRACK+IKE_FRAG_ALLOW {using isakmp#1 msgid:effe9287 proposal=AES(12)_256-SHA1(2)_000 pfsgroup=no
-pfs}
Apr  3 12:44:28: "example-one" #2: ignoring informational payload IPSEC_RESPONDER_LIFETIME, msgid=effe9287, length=28
Apr  3 12:44:28: | ISAKMP Notification Payload
Apr  3 12:44:28: |   00 00 00 1c  00 00 00 01  03 04 60 00
Apr  3 12:44:28: "example-one" #2: transition from state STATE_QUICK_I1 to state STATE_QUICK_I2
Apr  3 12:44:28: "example-one" #2: STATE_QUICK_I2: sent QI2, IPsec SA established tunnel mode {ESP/NAT=>0x414c5406 <0x8df53642 xfrm=AES_256-HMAC_SHA1 NATOA=none NATD=144.230.xxx.xxx:4500 DPD=passive} 

Below is the tcpdump.

# tcpdump -n -i eth0 esp or udp port 500 or udp port 4500
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:58:42.229262 IP 10.0.10.26.ipsec-nat-t > 144.230.xxx.xxx.ipsec-nat-t: isakmp-nat-keep-alive
11:58:42.229280 IP 10.0.10.26.ipsec-nat-t > 144.230.xxx.xxx.ipsec-nat-t: isakmp-nat-keep-alive
11:58:44.487779 IP 144.230.xxx.xxx.ipsec-nat-t > 10.0.10.26.ipsec-nat-t: NONESP-encap: isakmp: phase 2/others ? inf[E]
11:58:44.487986 IP 10.0.10.26.ipsec-nat-t > 144.230.xxx.xxx.ipsec-nat-t: NONESP-encap: isakmp: phase 2/others ? inf[E]

And below is sysctl command output.

sysctl -p
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.eth0.rp_filter = 0
net.ipv4.conf.lo.rp_filter = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.eth0.send_redirects = 0
net.ipv4.conf.lo.send_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.eth0.accept_redirects = 0
net.ipv4.conf.lo.accept_redirects = 0
net.ipv4.ip_forward = 1 

Below are the iptable rule applied on VPN server.

 iptables -t nat --line-numbers -L
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination        
1    DNAT       all  --  anywhere             ec2-107-23-xxx-xxx.compute-1.amazonaws.com  to:10.0.10.20

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination        

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination        
1    SNAT       all  --  anywhere             ip-10-0-10-20.ec2.internal  to:107.23.xxx.xxx
2    MASQUERADE  all  --  anywhere             anywhere

iptables -t nat -A PREROUTING -d 107.23.xxx.xxx -j DNAT --to-destination 10.0.10.20
iptables -t nat -A POSTROUTING -d 10.0.10.20 -j SNAT --to-source 107.23.xxx.xxx

Solution

Below are the steps to get this working.

1. You need to update the route table with interface ID of your VPN Server. So that all traffic from your FTP Server reach the right subnet via VPN Host i.e {144.226.xxx.xxx/32 eniXXXXXX(interface id of your VPN Server)}

2. IPSEC configuration would be like below

conn test
  authby=secret
  auto=start
  type=tunnel
  left=%defaultroute
  leftid=10.0.10.30 #### Private IP of your VPN Server
  leftsubnet=107.23.xx.xxx/32 ### Public IP of FTP Server
  leftnexthop=%defaultroute
  right=144.230.xxx.xxx ### Peer IP of Cisco Device
  rightid=144.230.xxx.xxx ### Peer IP of Cisco Device
  rightnexthop=107.23.XXX.XXX ### E IP of your VPN Server
  rightsubnet=144.226.xxx.xxx/32 ### Right/Client Side Subnet
  keyexchange=ike
  ike=aes256-sha1;modp1024
  phase2=esp
  phase2alg=aes256-sha1;modp1024
  aggrmode=no
  pfs=no

3. Finally you need to add nat rules in your firewall.

   iptables -t nat -A PREROUTING -d 107.23.xxx.xxx (FTP Server IP) -jDNAT --to-destination 10.0.10.32 (Private ip of your FTP Server)

   iptables -t nat -A POSTROUTING -s 10.0.10.32 -d 144.26.XXX.XXX (Client/Right side IPs) -j SNAT --to-source 107.23.XXX.XXX (FTP Server IP)

Note:

1. IPv4 forwarding should be enabled in sysctl.conf.
2. In the secret file use your private ip i.e "10.0.10.30(VPN host private IP) 144.23.xxx.xxx (Cisco Peer IP) : "

Comments

[Admin]

Well... how is it failing? What's in the log?

[Admin]

@Lenniey Please have a look. I have updated question with more information.

[Admin]

Your tunnel is being established, so I don't think it's a IPSec related problem. You have to check your routing. Why do you SNAT the local IP destination traffic to your public IP on the VPN server?

[Admin]

@Lenniey If I do not SNAT, tunnel doesn't get establish between VPN server and CISCO 3000

[Admin]

I believe @Lenniey is correct, and that is not the correct solution. The AWS Internet Gateway does translation for you and it looks like you are (maybe?) trying to undo it in a "two wrongs make a right" attempt. Instead, remove that. Next, assumung EC2 is the "left" side, try left=ec2-private-ip leftid=ec2-public-ip leftsourceip=ec2-private-ip (left and leftsourceip are private, and leftid is public). I have a couple of similar setups and this reflects my configuration.

[Admin]

@Michael-sqlbot we are not using IGW to reach peer IP of the client. We are using Elastic IP assigned/tied to my vpn server

[Admin]

I believe you are, in fact, using the Internet Gateway -- not to be confused with a NAT Gateway (which is something entirely different, and doesn't support IPSec). EIPs don't work at all unless your subnet's default route points to the Internet Gateway (igw-xxxxxxxx), which does the static translation between public and private IPv4 for EC2 instances.

[Admin]

Yes You're right. Somehow I skipped it. Can you guide me how to set it up from scratch or if you've any link I can refer to. I have my current IP whitelisted at client side. Is there any way I can use the same IP for IPSec. Another thing is IGW will use dynamic IP. So any suggestions based on above scenario. Thanks in advance for guiding me.