Cisco – Why do Cisco IOS routers hang in the middle of large downloads

ciscoiosrouter

After a few years in use. We have seen Cisco 871 and 851 routers that would hang if you had a single download that was more than 100M large. It is intermittent. Sometimes the problem goes away, sometimes it happens on very small downloads (just a 10KB web page). It seems that the just about all the downloads eventually finish, but the bigger the download the longer the hang.

Is there a way to resolve this? (short of router replacement which is what we have been doing)

We are revisiting this on a Cisco 851 that is one year and two months old. At this point, similar hangs seem to be occurring, at a much less important scale. In this case, the customer has purchased a 30Mbps up/down internet connection, and they are only able to get 5Mbps/20Mbps up/down. At times, download speed is reduced to 5Mbps.

I will attempt what has already been suggested below next time I am out there (hopefully next week) and edit in my findings.

I an ACL on Vlan1 and on the Fa4. I also have a few ACLs that were replaced and are not used. The ACLs are about 45 lines and about half the lines are remarks. I have posted the config below. Personal information is masked with words such as WAN IP or hostname HIDDEN

If you have suggestions such as performance improvements for the configuration code, or information such as whether I can expect 30Mbps on an 851, that would be appreciated.

Current configuration : 18157 bytes
!
version 12.4
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers
!
hostname HIDDEN
!
boot-start-marker
boot-end-marker
!
logging buffered 51200
logging console critical
enable secret 5 --GIBBERISH---
!
aaa new-model
!
!         
aaa authentication login local_authen local
aaa authorization exec local_author local 
!
!
aaa session-id common
clock timezone EST -5
!
crypto pki trustpoint TP-self-signed-4140887523
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-4140887523
 revocation-check none
 rsakeypair TP-self-signed-4140887523
!
!
dot11 syslog
no ip source-route
no ip dhcp use vrf connected
ip dhcp binding cleanup interval 60
ip dhcp excluded-address 10.10.10.1
ip dhcp excluded-address 192.168.1.1
!
ip dhcp pool ccp-pool
   import all
   network 10.10.10.0 255.255.255.248
   default-router 10.10.10.1 
   lease 0 2
!
ip dhcp pool sdm-pool1
   import all
   network 192.168.1.0 255.255.255.0
   dns-server --DNS Server 1-- --DNS Server 2-- 
   default-router 192.168.1.1 
!
!
ip cef
ip inspect name DEFAULT100 appfw DEFAULT100
ip inspect name DEFAULT100 cuseeme
ip inspect name DEFAULT100 ftp
ip inspect name DEFAULT100 h323
ip inspect name DEFAULT100 icmp
ip inspect name DEFAULT100 rcmd
ip inspect name DEFAULT100 realaudio
ip inspect name DEFAULT100 rtsp
ip inspect name DEFAULT100 esmtp
ip inspect name DEFAULT100 sqlnet
ip inspect name DEFAULT100 streamworks
ip inspect name DEFAULT100 tftp
ip inspect name DEFAULT100 tcp
ip inspect name DEFAULT100 udp
ip inspect name DEFAULT100 vdolive
ip inspect name DEFAULT100 https
ip inspect name DEFAULT100 dns
no ip bootp server
no ip domain lookup
ip domain name noexist.example.com
ip name-server --DNS Server 2--
ip name-server --DNS Server 1--
!
appfw policy-name DEFAULT100
  application im aol
    service default action reset 
    service text-chat action reset 
    server deny name login.oscar.aol.com
    server deny name toc.oscar.aol.com
    server deny name oam-d09a.blue.aol.com
  application im msn
    service default action reset 
    service text-chat action reset 
    server deny name messenger.hotmail.com
    server deny name gateway.messenger.hotmail.com
    server deny name webmessenger.msn.com
  application http
    port-misuse im action reset alarm
  application im yahoo
    service default action reset 
    service text-chat action reset 
    server deny name scs.msg.yahoo.com
    server deny name scsa.msg.yahoo.com
    server deny name scsb.msg.yahoo.com
    server deny name scsc.msg.yahoo.com
    server deny name scsd.msg.yahoo.com
    server deny name messenger.yahoo.com
    server deny name cs16.msg.dcn.yahoo.com
    server deny name cs19.msg.dcn.yahoo.com
    server deny name cs42.msg.dcn.yahoo.com
    server deny name cs53.msg.dcn.yahoo.com
    server deny name cs54.msg.dcn.yahoo.com
    server deny name ads1.vip.scd.yahoo.com
    server deny name radio1.launch.vip.dal.yahoo.com
    server deny name in1.msg.vip.re2.yahoo.com
    server deny name data1.my.vip.sc5.yahoo.com
    server deny name address1.pim.vip.mud.yahoo.com
    server deny name edit.messenger.yahoo.com
    server deny name http.pager.yahoo.com
    server deny name privacy.yahoo.com
    server deny name csa.yahoo.com
    server deny name csb.yahoo.com
    server deny name csc.yahoo.com
!
!
!
username surfn privilege 15 secret 5 $1$1hrm$0yfIN0jK56rOm9cXfm2a21
! 
!
archive
 log config
  hidekeys
!
!
ip tcp synwait-time 10
ip ssh time-out 60
ip ssh authentication-retries 2
!
!
!         
interface Null0
 no ip unreachables
!
interface FastEthernet0
!
interface FastEthernet1
!
interface FastEthernet2
!
interface FastEthernet3
!
interface FastEthernet4
 description $ES_WAN$$FW_OUTSIDE$
 ip address --WAN IP-- 255.255.255.0
 ip access-group 123 in
 ip verify unicast reverse-path
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip inspect DEFAULT100 out
 ip nat outside
 ip virtual-reassembly
 ip route-cache flow
 duplex auto
 speed auto
!
interface Vlan1
 description $ETH-SW-LAUNCH$$INTF-INFO-HWIC 4ESW$$ES_LAN$$FW_INSIDE$
 ip address 192.168.1.1 255.255.255.0
 ip access-group 102 in
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat inside
 ip virtual-reassembly
 ip route-cache flow
 ip tcp adjust-mss 1452
!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 --ISP Gateway--
!
ip http server
ip http access-class 2
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
ip nat inside source list 1 interface FastEthernet4 overload
!
logging trap debugging
access-list 1 remark Telnet, SSH access
access-list 1 permit 192.168.1.0 0.0.0.255
access-list 1 deny   any
access-list 2 remark HTTP, HTTPS access
access-list 2 permit 192.168.1.0 0.0.0.255
access-list 2 deny   any
access-list 101 HIDDEN
access-list 102 HIDDEN
access-list 121 HIDDEN
access-list 122 HIDDEN
access-list 123 HIDDEN
no cdp run
!
control-plane
!
banner exec ^C
% Password expiration warning.
-----------------------------------------------------------------------

Cisco Configuration Professional (Cisco CP) is installed on this device 
and it provides the default username "cisco" for  one-time use. If you have 
already used the username "cisco" to login to the router and your IOS image 
supports the "one-time" user option, then this username has already expired. 
You will not be able to login to the router with this username after you exit 
this session.

It is strongly suggested that you create a new username with a privilege level 
of 15 using the following command.

username <myuser> privilege 15 secret 0 <mypassword>

Replace <myuser> and <mypassword> with the username and password you 
want to use.

-----------------------------------------------------------------------
^C
banner login ^CCAuthorized access only!
 Disconnect IMMEDIATELY if you are not an authorized user!^C
!
line con 0
 login authentication local_authen
 no modem enable
 transport output telnet
line aux 0
 login authentication local_authen
 transport output telnet
line vty 0 4
 access-class 100 in
 privilege level 15
 authorization exec local_author
 login authentication local_authen
 transport input telnet ssh
!
scheduler max-task-time 5000
scheduler allocate 4000 1000
scheduler interval 500
end

Best Answer

George, I'm seeing the following message:

%FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:3558911335 1500 bytes is out-of-order; expected seq:3558888055. Reason: TCP reassembly queue overflow - session 192.168.23.38:54435 to 65.199.63.58:801024

The following command seems to have worked for me by extending the queue reassembly queue.

ip inspect tcp reassembly queue length 1024

I suppose it's a long shot, since I don't know your config. Hope that helps!

Colin Jaccino