Randomly unable to connect to localhost web server

apache-2.2iis-6networkingwindows-server-2003

We've got a very strange issue that cropped up a few weeks ago and have been unable to resolve.

We are running a couple of web sites in IIS (port 80,443) and in Apache (8080,8090) all on the same Windows Server 2003 SP2 machine. We've been running this configuration for a couple of years now.

The web applications running in IIS connect sometimes connect to the applications running in Apache (on the same server) before responding to the client. Other times the applications will connect to a database server running on another server, and sometimes they will connect to a Windows file share on another server as well.

In all three of the above scenarios we will get the application sporadically reporting one of the following errors:

  • Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.
  • The underlying connection was closed: An unexpected error occurred on a receive.

In addition, we've noticed that while logged into the server while the problem is occurring, attempting to do a request to http://localhost/etc, http://127.0.0.1/etc, http://192.168.xxx.xxx/etc (local IP) all will give a "Connection was reset" error message (Firefox). Both IIS and Apache web requests fail. We are able to connect to the server from a different machine (using IP address or hostname), and we can connect to external sites from the server and doing a ping to itself does not drop out during that time period.

The problem will magically correct itself for a random period of time. Sometimes we can go over 24 hours with out a problem, other times just 20-30 minutes. While the problem is occurring it can last from a few seconds to several minutes (usually no more than 10-15).

We've also experienced no problems connecting to the database server or file share server from other servers at times when we experience it from this server.

Any ideas as to where we should be looking?

Update:
So we're still getting this areas, but to add some more detail we get these errors randomly on connections to multiple servers and several different types of connections. We get it on cifs (File Sharing), SQL Server, and web connections to multiple servers both on the LAN and WAN, and to itself. Most of the time it is the "An established connection was aborted by the software in your host machine."

NETSH DUMP

#========================
# Interface configuration
#========================
pushd interface

reset all


popd
# End of interface configuration

#========================
# Interface configuration
#========================
pushd interface ipv6

uninstall


popd
# End of interface configuration



# ----------------------------------
# ISATAP Configuration
# ----------------------------------
pushd interface ipv6 isatap



popd
# End of ISATAP configuration



# ----------------------------------
# 6to4 Configuration
# ----------------------------------
pushd interface ipv6 6to4

reset



popd
# End of 6to4 configuration

#========================
# Port Proxy configuration
#========================
pushd interface portproxy

reset


popd
# End of Port Proxy configuration



# ---------------------------------- 
# Interface IP Configuration         
# ---------------------------------- 
pushd interface ip


# Interface IP Configuration for "SW-1A"

set address name="SW-1A" source=static addr=192.168.xxx.51 mask=255.255.255.0
add address name="SW-1A" addr=192.168.xxx.50 mask=255.255.255.0
set address name="SW-1A" gateway=192.168.xxx.254 gwmetric=0
set dns name="SW-1A" source=static addr=192.168.xxx.2 register=PRIMARY
add dns name="SW-1A" addr=192.168.xxx.3 index=2
set wins name="SW-1A" source=static addr=none


popd
# End of interface IP configuration


# ------------------------------------
# Bridge configuration (not supported)
# ------------------------------------

# ------------------------------------
# End of Bridge configuration
# ------------------------------------


# ----------------------------------------- 
# aaaa Configuration                         
# ----------------------------------------- 
# This script will NOT work across different versions of IAS.
# ----------------------------------------- 

# aaaa configuration script.  
# Known Issues and limitations: 
# Import/Export between different versions is not supported.
# IAS.MDB Version = 7
pushd aaaa
set config  blob=\
blob snippped
\
AA\
*\
A7ACI/wD\
\
*\
A7ACI/wD\
\
*\
A7ACI/wD\
\
*\
A7ACI/wD\
\
*\
A7ACI/wD\
\
*\
A7ACI/wD\
\
*\
A7ACI/wD\
\
*\
A7ACI/wD\
\
*\
A7ACI/wD\
\
*\
A7ACI/wD\
\
*\
A7ACI/wD\
\
*\
A7ACI/wD\
\
*\
A7ACI/wD\
\
*\
A7ACI/wD\
\
*
popd

# End of aaaa show config


# End of aaaa configuration.                  




# ----------------------------------------- 
# Remote Access Configuration               
# ----------------------------------------- 
pushd ras

set authmode mode = standard
delete authtype type = PAP
delete authtype type = SPAP
delete authtype type = MD5CHAP
delete authtype type = MSCHAP
delete authtype type = MSCHAPv2
delete authtype type = EAP
add authtype type = MSCHAP
add authtype type = MSCHAPv2
delete link type = SWC
delete link type = LCP
add link type = SWC
add link type = LCP
delete multilink type = MULTI
delete multilink type = BACP
add multilink type = MULTI
add multilink type = BACP

set user name = ASPNET dialin = policy cbpolicy = none 
set user name = Guest dialin = policy cbpolicy = none 
set user name = IUSR_WX-WWW1 dialin = policy cbpolicy = none 
set user name = IWAM_WX-WWW1 dialin = policy cbpolicy = none 
set user name = customuser1 dialin = policy cbpolicy = none 
set user name = customuser2 dialin = policy cbpolicy = none 
set user name = customuser3 dialin = policy cbpolicy = none 
set user name = SUPPORT_388945a0 dialin = policy cbpolicy = none 
set user name = customuser4 dialin = policy cbpolicy = none 


popd

# End of Remote Access configuration.        




# ----------------------------------------- 
# Remote Access AppleTalk Configuration     
# ----------------------------------------- 
pushd ras appletalk

set negotiation mode = allow

popd

# End of Remote Access AppleTalk Configuration. 



# ----------------------------------------- 
# Remote Access Diagnostics Configuration   
# ----------------------------------------- 
pushd ras diagnostics

set rastracing component = * state = disabled

set modemtracing state = disabled

set cmtracing state = disabled

set securityeventlogs state = disabled


popd

# End of Remote Access Diagnostics Configuration.




# ----------------------------------------- 
# Remote Access IP Configuration            
# ----------------------------------------- 
pushd ras ip

delete pool

set negotiation mode = allow
set access mode = all
set addrreq mode = deny
set broadcastnameresolution mode = disabled
set addrassign method = auto

popd

# End of Remote Access IP configuration.     



# ----------------------------------------- 
# Remote Access IPX Configuration           
# ----------------------------------------- 
pushd ras ipx

set negotiation mode = allow
set access mode = all
set nodereq mode = allow
set netassign method = autosame

popd

# End of Remote Access IPX configuration.    




# ----------------------------------------- 
# Remote Access NBF Configuration           
# ----------------------------------------- 
pushd ras netbeui

set negotiation mode = allow
set access mode = all

popd

# End of Remote Access NBF configuration.   




# ----------------------------------------- 
# Remote Access AAAA Configuration          
# ----------------------------------------- 
pushd ras aaaa

set authentication provider = windows
set accounting provider = windows

delete authserver name = *
delete acctserver name = *



popd

# End of Remote Access AAAA configuration.     


# Routing Configuration
pushd routing
reset
popd
# IP Configuration
pushd routing ip
reset
set loglevel error
add preferenceforprotocol proto=LOCAL preflevel=1
add preferenceforprotocol proto=NetMgmt preflevel=10
add preferenceforprotocol proto=STATIC preflevel=3
add preferenceforprotocol proto=NONDOD preflevel=5
add preferenceforprotocol proto=AUTOSTATIC preflevel=7
add preferenceforprotocol proto=OSPF preflevel=110
add preferenceforprotocol proto=RIP preflevel=120
add interface name="SW-1B" state=enable
set filter name="SW-1B" fragcheck=disable
add interface name="SW-1A" state=enable
set filter name="SW-1A" fragcheck=disable
add interface name="Internal" state=enable
set filter name="Internal" fragcheck=disable
add interface name="Loopback" state=enable
set filter name="Loopback" fragcheck=disable
popd
# End of IP configuration



# ---------------------------------- 
# DNS Proxy configuration            
# ---------------------------------- 
pushd routing ip dnsproxy
uninstall


popd
# End of DNS proxy configuration



# ---------------------------------- 
# IGMP Configuration                 
# ---------------------------------- 
pushd routing ip igmp
uninstall


popd
# End of IGMP configuration



# ---------------------------------- 
# NAT configuration                  
# ---------------------------------- 
pushd routing ip nat
uninstall


popd




# ---------------------------------- 
# OSPF configuration                 
# ---------------------------------- 

pushd routing ip ospf
uninstall

popd
# End of OSPF configuration




# ---------------------------------- 
# DHCP Relay Agent configuration     
# ---------------------------------- 
pushd routing ip relay
uninstall


popd
# End of DHCP Relay configuration



# ---------------------------------- 
# RIP configuration                  
# ---------------------------------- 
pushd routing ip rip
uninstall


popd
# End of RIP configuration



# ---------------------------------- 
# Router Discovery Configuration     
# ---------------------------------- 
pushd routing ip routerdiscovery
uninstall
add interface name="SW-1B" disc=disable minint=7 maxint=10 life=30 level=0
add interface name="SW-1A" disc=disable minint=7 maxint=10 life=30 level=0
add interface name="Internal" disc=disable minint=7 maxint=10 life=30 level=0
add interface name="Loopback" disc=disable minint=7 maxint=10 life=30 level=0


popd


# ---------------------------------- 
# DHCP Allocator Configuration       
# ---------------------------------- 
pushd routing ip autodhcp
uninstall


popd
# End of DHCP Allocator Configuration


Loading of DLL WinsEvnt.dll failed.
Wins Operation failed with Error There are no more endpoints available from the endpoint mapper.

Update:
We ended up installing Windows Server 2008 R2 on the same hardware in late July and the problem went away and we've not looked back since. There's a point were you just cut your losses, bite the bullet and run with it.

Best Answer

One possibility: ephemeral port exhaustion. Try something like netstat -an | find /c ":" to count how many connections you have in all the various states. If that number is over ten thousand or so, then chances are that this is your issue.