I've got an Apache 2.0.52 server on CentOS 4 that front-ends a couple of App servers (mix of Jetty and Tomcat). Apache has a handful of virtual hosts configured like this:
<VirtualHost www1.example.com:443>
ServerName www1.example.com
DocumentRoot "/mnt/app_web/html"
SSLEngine on
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP
SSLCertificateFile /etc/httpd/conf/ssl.crt/server.crt
SSLCertificateChainFile /etc/httpd/conf/ssl.crt/chain.crt
SSLCertificateKeyFile /etc/httpd/conf/ssl.key/server.key
SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
RewriteEngine on
RewriteRule ^/app1/(.*)$ http://app1.example.com:8080/app1/$1 [P,L]
RewriteRule ^/app2/(.*)$ http://app2.example.com:8080/app2/$1 [P,L]
</VirtualHost>
However, I'm getting the following errors in the logs intermittently:
[Fri Dec 04 07:19:41 2009] [error] (113)No route to host: proxy: HTTP: attempt to connect to 10.0.0.1:8080 (app1.example.com) failed
I initially tried turning off IPv6, and that seemed to largely cure it, but I still have sporadic bursts of these messages.
Additionally, we're running memcache on same front-end and during the times when I'm getting those messages in Apache's log, the following command doesn't work:
echo stats | nc 127.0.0.1 11211
No messages are printed, but neither are the stats printed. I am completely lost as to how to proceed with troubleshooting this. =(
Best Answer
To solve this problem you need to add rule(s) in the 'iptables' of your App servers. For Red Hat Enterprise the file is '/etc/sysconfig/iptables' . It should be the same for CentOS.
You probaly have one or more rules that accept NEW connection from the front-ends that look like these:
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s 'IP of the front-ends' --dport 'port number' -j ACCEPT
OR
-A RH-Firewall-1-INPUT -m state --state NEW -m multiport -m tcp -p tcp -s 'IP of the front-ends' --dports 'ports numbers' -j ACCEPT
Your problem shoud be solved by adding rules that send a tcp-reset to the front-ends for each SYN packet that passed throught the preceedings rules. The rules should looks like these:
-A RH-Firewall-1-INPUT -m tcp -p tcp -s 'IP of the front-ends' --dport 'port number' --syn -j REJECT --reject-with tcp-reset
OR
-A RH-Firewall-1-INPUT -m multiport -m tcp -p tcp -s 'IP of the front-ends' --dports 'ports numbers' --syn -j REJECT --reject-with tcp-reset
Add the rules near the end of your 'iptables' just before the rule that looks like:
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
Good luck.
Paul