Why do Cisco IOS routers hang in the middle of large downloads?

Question

After a few years in use. We have seen Cisco 871 and 851 routers that would hang if you had a single download that was more than 100M large. It is intermittent. Sometimes the problem goes away, sometimes it happens on very small downloads (just a 10KB web page). It seems that the just about all the downloads eventually finish, but the bigger the download the longer the hang.

Is there a way to resolve this? (short of router replacement which is what we have been doing)

We are revisiting this on a Cisco 851 that is one year and two months old. At this point, similar hangs seem to be occurring, at a much less important scale. In this case, the customer has purchased a 30Mbps up/down internet connection, and they are only able to get 5Mbps/20Mbps up/down. At times, download speed is reduced to 5Mbps.

I will attempt what has already been suggested below next time I am out there (hopefully next week) and edit in my findings.

I an ACL on Vlan1 and on the Fa4. I also have a few ACLs that were replaced and are not used. The ACLs are about 45 lines and about half the lines are remarks. I have posted the config below. Personal information is masked with words such as WAN IP or hostname HIDDEN

If you have suggestions such as performance improvements for the configuration code, or information such as whether I can expect 30Mbps on an 851, that would be appreciated.

Current configuration : 18157 bytes
!
version 12.4
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers
!
hostname HIDDEN
!
boot-start-marker
boot-end-marker
!
logging buffered 51200
logging console critical
enable secret 5 --GIBBERISH---
!
aaa new-model
!
!         
aaa authentication login local_authen local
aaa authorization exec local_author local 
!
!
aaa session-id common
clock timezone EST -5
!
crypto pki trustpoint TP-self-signed-4140887523
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-4140887523
 revocation-check none
 rsakeypair TP-self-signed-4140887523
!
!
dot11 syslog
no ip source-route
no ip dhcp use vrf connected
ip dhcp binding cleanup interval 60
ip dhcp excluded-address 10.10.10.1
ip dhcp excluded-address 192.168.1.1
!
ip dhcp pool ccp-pool
   import all
   network 10.10.10.0 255.255.255.248
   default-router 10.10.10.1 
   lease 0 2
!
ip dhcp pool sdm-pool1
   import all
   network 192.168.1.0 255.255.255.0
   dns-server --DNS Server 1-- --DNS Server 2-- 
   default-router 192.168.1.1 
!
!
ip cef
ip inspect name DEFAULT100 appfw DEFAULT100
ip inspect name DEFAULT100 cuseeme
ip inspect name DEFAULT100 ftp
ip inspect name DEFAULT100 h323
ip inspect name DEFAULT100 icmp
ip inspect name DEFAULT100 rcmd
ip inspect name DEFAULT100 realaudio
ip inspect name DEFAULT100 rtsp
ip inspect name DEFAULT100 esmtp
ip inspect name DEFAULT100 sqlnet
ip inspect name DEFAULT100 streamworks
ip inspect name DEFAULT100 tftp
ip inspect name DEFAULT100 tcp
ip inspect name DEFAULT100 udp
ip inspect name DEFAULT100 vdolive
ip inspect name DEFAULT100 https
ip inspect name DEFAULT100 dns
no ip bootp server
no ip domain lookup
ip domain name noexist.example.com
ip name-server --DNS Server 2--
ip name-server --DNS Server 1--
!
appfw policy-name DEFAULT100
  application im aol
    service default action reset 
    service text-chat action reset 
    server deny name login.oscar.aol.com
    server deny name toc.oscar.aol.com
    server deny name oam-d09a.blue.aol.com
  application im msn
    service default action reset 
    service text-chat action reset 
    server deny name messenger.hotmail.com
    server deny name gateway.messenger.hotmail.com
    server deny name webmessenger.msn.com
  application http
    port-misuse im action reset alarm
  application im yahoo
    service default action reset 
    service text-chat action reset 
    server deny name scs.msg.yahoo.com
    server deny name scsa.msg.yahoo.com
    server deny name scsb.msg.yahoo.com
    server deny name scsc.msg.yahoo.com
    server deny name scsd.msg.yahoo.com
    server deny name messenger.yahoo.com
    server deny name cs16.msg.dcn.yahoo.com
    server deny name cs19.msg.dcn.yahoo.com
    server deny name cs42.msg.dcn.yahoo.com
    server deny name cs53.msg.dcn.yahoo.com
    server deny name cs54.msg.dcn.yahoo.com
    server deny name ads1.vip.scd.yahoo.com
    server deny name radio1.launch.vip.dal.yahoo.com
    server deny name in1.msg.vip.re2.yahoo.com
    server deny name data1.my.vip.sc5.yahoo.com
    server deny name address1.pim.vip.mud.yahoo.com
    server deny name edit.messenger.yahoo.com
    server deny name http.pager.yahoo.com
    server deny name privacy.yahoo.com
    server deny name csa.yahoo.com
    server deny name csb.yahoo.com
    server deny name csc.yahoo.com
!
!
!
username surfn privilege 15 secret 5 $1$1hrm$0yfIN0jK56rOm9cXfm2a21
! 
!
archive
 log config
  hidekeys
!
!
ip tcp synwait-time 10
ip ssh time-out 60
ip ssh authentication-retries 2
!
!
!         
interface Null0
 no ip unreachables
!
interface FastEthernet0
!
interface FastEthernet1
!
interface FastEthernet2
!
interface FastEthernet3
!
interface FastEthernet4
 description $ES_WAN$$FW_OUTSIDE$
 ip address --WAN IP-- 255.255.255.0
 ip access-group 123 in
 ip verify unicast reverse-path
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip inspect DEFAULT100 out
 ip nat outside
 ip virtual-reassembly
 ip route-cache flow
 duplex auto
 speed auto
!
interface Vlan1
 description $ETH-SW-LAUNCH$$INTF-INFO-HWIC 4ESW$$ES_LAN$$FW_INSIDE$
 ip address 192.168.1.1 255.255.255.0
 ip access-group 102 in
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat inside
 ip virtual-reassembly
 ip route-cache flow
 ip tcp adjust-mss 1452
!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 --ISP Gateway--
!
ip http server
ip http access-class 2
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
ip nat inside source list 1 interface FastEthernet4 overload
!
logging trap debugging
access-list 1 remark Telnet, SSH access
access-list 1 permit 192.168.1.0 0.0.0.255
access-list 1 deny   any
access-list 2 remark HTTP, HTTPS access
access-list 2 permit 192.168.1.0 0.0.0.255
access-list 2 deny   any
access-list 101 HIDDEN
access-list 102 HIDDEN
access-list 121 HIDDEN
access-list 122 HIDDEN
access-list 123 HIDDEN
no cdp run
!
control-plane
!
banner exec ^C
% Password expiration warning.
-----------------------------------------------------------------------

Cisco Configuration Professional (Cisco CP) is installed on this device 
and it provides the default username "cisco" for  one-time use. If you have 
already used the username "cisco" to login to the router and your IOS image 
supports the "one-time" user option, then this username has already expired. 
You will not be able to login to the router with this username after you exit 
this session.

It is strongly suggested that you create a new username with a privilege level 
of 15 using the following command.

username <myuser> privilege 15 secret 0 <mypassword>

Replace <myuser> and <mypassword> with the username and password you 
want to use.

-----------------------------------------------------------------------
^C
banner login ^CCAuthorized access only!
 Disconnect IMMEDIATELY if you are not an authorized user!^C
!
line con 0
 login authentication local_authen
 no modem enable
 transport output telnet
line aux 0
 login authentication local_authen
 transport output telnet
line vty 0 4
 access-class 100 in
 privilege level 15
 authorization exec local_author
 login authentication local_authen
 transport input telnet ssh
!
scheduler max-task-time 5000
scheduler allocate 4000 1000
scheduler interval 500
end

Solution 1

George, I'm seeing the following message:

%FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:3558911335 1500 bytes is out-of-order; expected seq:3558888055. Reason: TCP reassembly queue overflow - session 192.168.23.38:54435 to 65.199.63.58:801024

The following command seems to have worked for me by extending the queue reassembly queue.

ip inspect tcp reassembly queue length 1024

I suppose it's a long shot, since I don't know your config. Hope that helps!

Colin Jaccino

Solution 2

How many users do you have behind these routers? Presumably you're doing NAT on a single, external address. Modern software, especially webservices like facebook chat, etc. open a lot of concurrent TCP connections. Cisco's, I believe, have a statically sized NAT translation table. It may be overflowing and evicting the oldest connection? I'm afraid that I cannot offer any advice on checking if the NAT tables are overflowing or not.

I would not be inclined to suspect the firmware, especially if its been working reasonably for years before. I would, however, suggest giving the interface statistics a quick double check. If you're seeing dropped, invalid, badrx checksum, etc. errors on an interface, then that may well be the source of your problem. Either failing hardware, insufficient electrical isolation, or something else. I've stopped counting how many 'cheap' 5 port 10/100 or gigabit switch's i've seen semi-fail and become inconsistent and erratic in the past 3-4 years due to bulging/exploding capacitors internally. A

  show interfaces counters errors

statement should identify any troublesome interfaces very quickly.

Good luck.

Comments

[tony roth]

describe hang, my gut feeling is that its not the router!

[jftuga]

Have you looked into upgrading the router's firmware? This may actually fix this kind of issue.

[700 Software]

jiftuga: I have not. That may fix the problem. I will look in to it. . tony: By hang I mean that the download ceases to progress during the time of the hang (zero bytes per second). It seems pretty clear that by plugging the PC into the router we get the hang and by changing the IP address of the PC and replacing the router with a network switch we do not get the hang. (We have also ensured that there were no other PCs or anything else plugged into the router during our testing) (Also ensured that the router had plenty of spare CPU and Memory during the hang)

[700 Software]

Actually upgrading the firmware will require me to devote some time. I have had a hard time getting this issue to happen when I want it to, but we do have an unused router that was replaced because of this issue. I hope I can get it to have the problem again. If I can then I will upgrade the firmware, and try again. ---- I could get assigned to that task much easier if I could get a promise of good results.