Inbound SIP calls through Cisco 881 NAT hang up after a few seconds

Question

I've recently moved to a Cisco 881 router for my WAN link. I was previously using a Cisco Linksys WAG320N as my modem/router/WiFi AP/NAT firewall. The WAG320N is now running in bridged mode, so it's simply acting as a modem with one of it's LAN ports connected to FE4 WAN on my Cisco 881.

The Cisco 881 get's a DHCP provided IP from my ISP. My LAN is part of default Vlan 1 (192.168.1.0/24). General internet connectivity is working great, I've managed to setup static NAT rules for my HTTP/HTTPS/SMTP/etc. services which are running on my LAN. I don't know whether it's worth mentioning that I've opted to use NVI NAT (ip nat enable as opposed to the traditional ip nat outside/ip nat inside) setup. My reason for this is that NVI allows NAT loopback from my LAN to the WAN IP and back in to the necessary server on the LAN.

I run an Asterisk 1.8 PBX on my LAN, which connects to a SIP provider on the internet. Both inbound and outbound calls through the old setup (WAG320N providing routing/NAT) worked fine. However, since moving to the Cisco 881, inbound calls drop after around 10 seconds, whereas outbound calls work fine.

The following message is logged on my Asterisk PBX:

[Dec  9 15:27:45] WARNING[27734]: chan_sip.c:3641 retrans_pkt: Retransmission timeout reached on transmission [email protected] for seqno 1 (Critical Response) -- See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 6528ms with no response
[Dec  9 15:27:45] WARNING[27734]: chan_sip.c:3670 retrans_pkt: Hanging up call [email protected] - no reply to our critical packet (see https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions).

(I know that this is quite a common issue - I've spend the best part of 2 days solid on this, trawling Google.)

I've done as I am told and checked https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions. Referring to the section "Other SIP requests" in the page linked above, I believe that the hangup to be caused by the ACK from my SIP provider not being passed back through NAT to Asterisk on my PBX.

I tried to ascertain this by dumping the packets on my WAN interface on the 881. I managed to obtain a PCAP dump of packets in/out of my WAN interface. Here's an example of an ACK being reveived by the router from my provider:

689 21.219999   193.x.x.x   188.x.x.x   SIP 502 Request: ACK sip:[email protected] | 

However a SIP trace on the Asterisk server show's that there are no ACK's received in response to the 200 OK from my PBX:

http://pastebin.com/wwHpLPPz

In the past, I have been strongly advised to disable any sort of SIP ALGs on routers and/or firewalls and the many posts regarding this issue on the internet seem to support this. However, I believe on Cisco IOS, the config command to disable SIP ALG is no ip nat service sip udp port 5060 however, this doesn't appear to help the situation. To confirm that config setting is set:

Router1#show running-config | include sip
no ip nat service sip udp port 5060

Another interesting twist: for a short period of time, I tried another provider. Luckily, my trial account with them is still available, so I reverted my Asterisk config back to the revision before I integrated with my current provider. I then dialled in to the DDI associated with the trial trunk and the call didn't get hung up and I didn't get the error above! To me, this points at the provider, however I know, like all providers do, will say "There's no issues with our SIP proxies - it's your firewall." I'm tempted to agree with this, as this issue was not apparent with the old WAG320N router when it was doing the NAT'ing.

I'm sure you'll want to see my running-config too:

!
! Last configuration change at 15:55:07 UTC Sun Dec 9 2012 by xxx
version 15.2
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
no service password-encryption
service sequence-numbers
!
hostname Router1
!
boot-start-marker
boot-end-marker
!
!
security authentication failure rate 10 log
security passwords min-length 6
logging buffered 4096
logging console critical
enable secret 4 xxx
!
aaa new-model
!
!
aaa authentication login local_auth local
!
!
!
!
!
aaa session-id common
!
memory-size iomem 10
!
crypto pki trustpoint TP-self-signed-xxx
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-xxx
 revocation-check none
 rsakeypair TP-self-signed-xxx
!
!
crypto pki certificate chain TP-self-signed-xxx
 certificate self-signed 01
    quit
no ip source-route
no ip gratuitous-arps
ip auth-proxy max-login-attempts 5
ip admission max-login-attempts 5
!
!
!
!
!
no ip bootp server
ip domain name dmz.merlin.local
ip domain list dmz.merlin.local
ip domain list merlin.local
ip name-server x.x.x.x
ip inspect audit-trail
ip inspect udp idle-time 1800
ip inspect dns-timeout 7
ip inspect tcp idle-time 14400
ip inspect name autosec_inspect ftp timeout 3600
ip inspect name autosec_inspect http timeout 3600
ip inspect name autosec_inspect rcmd timeout 3600
ip inspect name autosec_inspect realaudio timeout 3600
ip inspect name autosec_inspect smtp timeout 3600
ip inspect name autosec_inspect tftp timeout 30
ip inspect name autosec_inspect udp timeout 15
ip inspect name autosec_inspect tcp timeout 3600
ip cef
login block-for 3 attempts 3 within 3
no ipv6 cef
!
!
multilink bundle-name authenticated
license udi pid CISCO881-SEC-K9 sn 
!
!
username xxx privilege 15 secret 4 xxx
username xxx secret 4 xxx
!
!
!
!
!
ip ssh time-out 60
! 
!
!
!
!
!
!
!
!
interface FastEthernet0
 no ip address
!
interface FastEthernet1
 no ip address
!
interface FastEthernet2
 no ip address
!
interface FastEthernet3
 switchport access vlan 2
 no ip address
!
interface FastEthernet4
 ip address dhcp
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat enable
 duplex auto
 speed auto
!
interface Vlan1
 ip address 192.168.1.1 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat enable
!
interface Vlan2
 ip address 192.168.0.2 255.255.255.0
!
ip forward-protocol nd
ip http server
ip http access-class 1
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
!
no ip nat service sip udp port 5060
ip nat source list 1 interface FastEthernet4 overload
ip nat source static tcp x.x.x.x 80 interface FastEthernet4 80
ip nat source static tcp x.x.x.x 443 interface FastEthernet4 443
ip nat source static tcp x.x.x.x 25 interface FastEthernet4 25
ip nat source static tcp x.x.x.x 587 interface FastEthernet4 587
ip nat source static tcp x.x.x.x 143 interface FastEthernet4 143
ip nat source static tcp x.x.x.x 993 interface FastEthernet4 993
ip nat source static tcp x.x.x.x 1723 interface FastEthernet4 1723
!
!
logging trap debugging
logging facility local2
access-list 1 permit 192.168.1.0 0.0.0.255
access-list 1 permit 192.168.0.0 0.0.0.255
no cdp run
!
!
!
!
control-plane
!
!
banner motd 
Authorized Access only

!
line con 0
 login authentication local_auth
 length 0
 transport output all
line aux 0
 exec-timeout 15 0
 login authentication local_auth
 transport output all
line vty 0 1
 access-class 1 in
 logging synchronous
 login authentication local_auth
 length 0
 transport preferred none
 transport input telnet
 transport output all
line vty 2 4
 access-class 1 in
 login authentication local_auth
 length 0
 transport input ssh
 transport output all
!
!
end

...and, if it's of any use, here's my Asterisk SIP config:

[general]
context=default                 ; Default context for calls
allowoverlap=no                 ; Disable overlap dialing support. (Default is yes)
udpbindaddr=0.0.0.0             ; IP address to bind UDP listen socket to (0.0.0.0 binds to all)
                                ; Optionally add a port number, 192.168.1.1:5062 (default is port 5060)
tcpenable=no                    ; Enable server for incoming TCP connections (default is no)
tcpbindaddr=0.0.0.0             ; IP address for TCP server to bind to (0.0.0.0 binds to all interfaces)
                                ; Optionally add a port number, 192.168.1.1:5062 (default is port 5060)
srvlookup=yes                   ; Enable DNS SRV lookups on outbound calls
                                ; Note: Asterisk only uses the first host
                                ; in SRV records
                                ; Disabling DNS SRV lookups disables the
                                ; ability to place SIP calls based on domain
                                ; names to some other SIP users on the Internet
                                ; Specifying a port in a SIP peer definition or
                                ; when dialing outbound calls will supress SRV
                                ; lookups for that peer or call.
directmedia=no                  ; Don't allow direct RTP media between extensions (doesn't work through NAT)
externhost=<MY DYNDNS HOSTNAME> ; Our external hostname to resolve to IP and be used in NAT'ed packets
localnet=192.168.1.0/24         ; Define our local network so we know which packets need NAT'ing
qualify=yes                     ; Qualify peers by default
dtmfmode=rfc2833                ; Set the default DTMF mode
disallow=all                    ; Disallow all codecs by default
allow=ulaw                      ; Allow G.711 u-law
allow=alaw                      ; Allow G.711 a-law

; ----------------------
; SIP Trunk Registration
; ----------------------

; Orbtalk
register => <MY SIP PROVIDER USER NAME>:[email protected]/<MY DDI>       ; Main Orbtalk number

; ----------
; Trunks
; ----------

[orbtalk]                       ; Main Orbtalk trunk
type=peer
insecure=invite
host=sipgw3.orbtalk.co.uk
nat=yes
username=<MY SIP PROVIDER USER NAME>
defaultuser=<MY SIP PROVIDER USER NAME>
fromuser=<MY SIP PROVIDER USER NAME>
secret=xxx
context=inbound

I really don't know where to go with this. If anyone can help me find out why these calls are being dropped off, I'd be grateful if you could chime in! Please let me know if any further info is required.

UPDATE 10/12/2012 (1):

@GerryEgan suggested I add the following static NAT: ip nat source static udp 192.168.1.2 5060 interface FastEthernet 4 5060 Whilst this seems to fix things up ok as far as I can tell, it would leave me open to brute force attacks on 5060.

@GerryEgan also suggested an inspect rule: ip inspect name autosec_inspect sip ...and (config-if)#ip inspect autosec_inspect in on my WAN interface Even after clearing NAT NVI translations, this alone doesn't solve the issue.

Here's a debug ip nat sip from an inbound call that gets dropped: http://pastebin.com/3WHPcyDD

Note: If I try the no ip nat service sip udp port 5060 config setting mentioned above, I don't get any debug output from debug ip nat sip, so for now, I have the default ip nat service sip udp port 5060 in my config - hope I'm not confusing things with this.

To make the situation worse, my static NAT NVI rules seem to be getting messed up on a reload of the router :-(. That's a separate issue though, but may have some bearing on your responses.

UPDATE 10/12/2012 (2): When calls are hung up by Asterisk, I'm seeing the following in /var/log/syslog:

Dec 10 20:02:39 <PBX> asterisk[26620]: rc_avpair_new: unknown attribute 1490026597

UPDATE 11/12/2012:

@GerryEgan suggested adding:

access-list 101 permit udp X.X.X.X 0.0.0.255 host 1.2.3.4 eq 5060
access-list 101 deny   udp any any eq 5060
access-list 101 permit ip any any

and

interface FastEthernet4
ip access-group 101 in

I could see the theory so I was expecting this to work, but unfortunately not.

This is weird, as the WAG320N I have before handled SIP to the same provider, PBX, etc. with no configuration or NAT'ing required. This prompted me to rebuild the new 881, so for completeness, here's the current config:

!
! Last configuration change at 20:20:15 UTC Tue Dec 11 2012 by xxx
version 15.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname xxx
!
boot-start-marker
boot-end-marker
!
!
enable secret 4 xxxx
!
aaa new-model
!
!
aaa authentication login local_auth local
!
!
!
!
!
aaa session-id common
!
memory-size iomem 10
!
crypto pki trustpoint TP-self-signed-xxx
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-xxx
 revocation-check none
 rsakeypair TP-self-signed-xxx
!
!
crypto pki certificate chain TP-self-signed-xxx
 certificate self-signed 01
  xxx
    quit
ip gratuitous-arps
ip auth-proxy max-login-attempts 5
ip admission max-login-attempts 5
!
!
!
!
!
ip domain list dmz.xxx.local
ip domain list xxx.local
ip domain name dmz.xxx.local
ip name-server 192.168.1.x
ip cef
login block-for 3 attempts 3 within 3
no ipv6 cef
!
!
multilink bundle-name authenticated
license udi pid CISCO881-SEC-K9 sn xxx
!
!
username admin privilege 15 secret 4 xxx
username joe secret 4 xxx
!
!
!
!
!
ip ssh time-out 60
! 
!
!
!
!
!
!
!
!
interface FastEthernet0
 no ip address
!
interface FastEthernet1
 no ip address
!
interface FastEthernet2
 no ip address
!
interface FastEthernet3
 switchport access vlan 2
 no ip address
!
interface FastEthernet4
 ip address dhcp
 ip access-group 101 in
 ip nat enable
 duplex auto
 speed auto
!
interface Vlan1
 ip address 192.168.1.x 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat enable
!
interface Vlan2
 ip address 192.168.0.x 255.255.255.0
!
ip forward-protocol nd
ip http server
ip http access-class 1
ip http authentication local
ip http secure-server
!
!
ip nat source list 1 interface FastEthernet4 overload
ip nat source list 2 interface FastEthernet4 overload
ip nat source static tcp 192.168.1.x 1723 interface FastEthernet4 1723
!
!
access-list 1 permit 192.168.0.0 0.0.0.255
access-list 2 permit 192.168.1.0 0.0.0.255
access-list 101 permit udp 193.x.x.0 0.0.0.255 any eq 5060
access-list 101 deny   udp any any eq 5060
access-list 101 permit ip any any
!
!
!
!
control-plane
!
!
banner motd 
Authorized Access only

!
line con 0
 exec-timeout 15 0
 login authentication local_auth
line aux 0
 exec-timeout 15 0
 login authentication local_auth
line vty 0 4
 access-class 2 in
 login authentication local_auth
 length 0
 transport input all
!
!
end

You'll notice that I have used access-list 101 permit udp 193.x.x.0 0.0.0.255 any eq 5060 instead of access-list 101 permit udp X.X.X.X 0.0.0.255 host 1.2.3.4 eq 5060. The reason being, my WAN IP is provided by DHCP, so I have no set IP to enter after host. I hope this is a valid substitute(?).

Again, just to re-iterate - this SIP provider worked perfectly on my old 'home' Linksys WAG320N with no configuration of NAT, firewall. SIP ALG was disabled by default on that system.

Another point of interest - I'm unable to NAT PPTP inbound through this router at the moment. It appears that GRE traffic is not getting through. Whilst I know the two are not same, it could be the same underlying configuration issue that's stopping the router from NAT'ing the traffic for each of the protocols correctly - I could be barking up the wrong tree...

UPDATE 16/12/2012:

I've seen an improvement when commenting out localnet=192.168.1.0/24 in Asterisk's sip.conf; I'm able to receive inbound calls without them getting hung up when this setting is omitted.

I know this could be seen as a fix to the issue, it's doesn't explain why the PBX worked with this setting for ~6 months when it's was sat behind the WAG320N as the firewall/router. Bearing this in mind, has anyone got any more ideas? Commenting out the localnet directive doesn't sit right in my head and I'm reluctant to rely on this as a permanent solution when it worked so well with the old router.

Solution

I've seen issues on multiple 881's I've deployed with doing the nat overload and using the Interface. For the most part making the wan link a static IP and changing the IP nat overload command to that has fixed most of the issues with calls and even phones freezing. Good luck

Comments

[GerryEgan]

Hi I'm no SIP expert but I notice that your missing a static NAT for inbound UDP5060, can you add it and retest if you haven't already? Also have you tried turning on SIP inspection? "Ip inspect name autosec_inspect sip" then use "IP inspect autosec_inspect in" under your wan interface.

[JoeNyland]

@GerryEgan: Thanks. I've updated my original question with the results of the things you have suggested.

[Ashish Anand]

fixing localnet also did the trick for me. I had SIP <-> IAX2 <-> SIP and it broke only in one direction, after (drum roll) 20 secs. (I didn't comment it out, though. Just fixed it from the previous subnet 10.x.x.0 to my current 192.x.x.0)

[NickW]

Why did the PBX work with that setting behind the WAG320N? Probably some subtle difference in the way the 881 dealt with the SIP traffic vs the WGA (ALG or not). The only way to really tell is set up both and do some traffic dumps. NAT and SIP is 1 gigantic headache though.

[JoeNyland]

I appreciate your input. I've updated the question with my results from your suggestions above. Sorry I wasn't clear in my last update: To confirm, adding ip nat source static udp 192.168.1.2 5060 interface FastEthernet 4 5060 seems to fix the issue, however this leaves a hole in the firewall. I then removed this, then tried your suggestion of adding ip inspect name autosec_inspect sip and (config-if)#ip inspect autosec_inspect in however this didn't help the ACKs get NAT'ed back through correctly. Is there anything that would hinder the effectiveness of this setting?