I have a small network with about 8 linux servers, a Cisco 2600 router, and a Cisco 3500XL switch. The router and switch have been configured and working properly for years.
About 6 hours ago the time to establish a connection via certain protocols skyrocketed. Connecting to a server via SSH can take a couple minutes to establish. But once the connection is made, it works normally. Copying files via scp is fast as well, but making the initial connection takes forever. Same with telnet.
However, connection via HTTP or HTTPS are perfectly fine. They cruise along like normal. Also SFTP seems to be fine as well.
SNMP connections also seem to be affected. My Cacti monitoring server has stopped working properly with timeout errors in the logs. PHPSVR: Poller[0] Maximum runtime of 292 seconds exceeded for the Script Server. Exiting. It gets intermittent, but mostly failed results from the hosts, however it is fairly reliable in reporting the router and switch cpu and memory.
The strange thing about the snmp monitoring of the switch is that some switch ports continue to be reported and show up in the cacti graphs, and others stop about 6 hours ago. But logging into the switch CLI shows activity on those ports.
My colo facility claims they have no networking issues at this time. I do think it is something in my network, but can't figure out what. I'm not seeing any significant traffic spikes and I can perform DNS queries without problems (dig queries taking around 34 msec). The delay happens even when connecting between hosts inside the network.
Any pointers on how to proceed troubleshooting this? I've reloaded both the switch and router and the problem continues. The following shows the output of ssh -vv with some comments that indicate how many seconds of a delay there were at different points in the login process.
https://gist.github.com/963682
Best Answer
Your name resolution has broken.