Apache reverse-proxy intermittent error 113 – No route to host

apache-2.2mod-proxymod-rewrite

I've got an Apache 2.0.52 server on CentOS 4 that front-ends a couple of App servers (mix of Jetty and Tomcat). Apache has a handful of virtual hosts configured like this:

<VirtualHost www1.example.com:443>
    ServerName www1.example.com
    DocumentRoot "/mnt/app_web/html"

    SSLEngine on
    SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP
    SSLCertificateFile      /etc/httpd/conf/ssl.crt/server.crt
    SSLCertificateChainFile /etc/httpd/conf/ssl.crt/chain.crt
    SSLCertificateKeyFile   /etc/httpd/conf/ssl.key/server.key
    SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0

    RewriteEngine on
    RewriteRule ^/app1/(.*)$ http://app1.example.com:8080/app1/$1 [P,L]
    RewriteRule ^/app2/(.*)$ http://app2.example.com:8080/app2/$1 [P,L]
</VirtualHost>

However, I'm getting the following errors in the logs intermittently:

[Fri Dec 04 07:19:41 2009] [error] (113)No route to host: proxy: HTTP: attempt to connect to 10.0.0.1:8080 (app1.example.com) failed

I initially tried turning off IPv6, and that seemed to largely cure it, but I still have sporadic bursts of these messages.

Additionally, we're running memcache on same front-end and during the times when I'm getting those messages in Apache's log, the following command doesn't work:

echo stats | nc 127.0.0.1 11211

No messages are printed, but neither are the stats printed. I am completely lost as to how to proceed with troubleshooting this. =(

Best Answer

To solve this problem you need to add rule(s) in the 'iptables' of your App servers. For Red Hat Enterprise the file is '/etc/sysconfig/iptables' . It should be the same for CentOS.

You probaly have one or more rules that accept NEW connection from the front-ends that look like these:

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s 'IP of the front-ends' --dport 'port number' -j ACCEPT

OR

-A RH-Firewall-1-INPUT -m state --state NEW -m multiport -m tcp -p tcp -s 'IP of the front-ends' --dports 'ports numbers' -j ACCEPT

Your problem shoud be solved by adding rules that send a tcp-reset to the front-ends for each SYN packet that passed throught the preceedings rules. The rules should looks like these:

-A RH-Firewall-1-INPUT -m tcp -p tcp -s 'IP of the front-ends' --dport 'port number' --syn -j REJECT --reject-with tcp-reset

OR

-A RH-Firewall-1-INPUT -m multiport -m tcp -p tcp -s 'IP of the front-ends' --dports 'ports numbers' --syn -j REJECT --reject-with tcp-reset

Add the rules near the end of your 'iptables' just before the rule that looks like:

-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited

Good luck.

Paul

Related Topic