Apache reverse-proxy intermittent error 113 - No route to host

Question

I've got an Apache 2.0.52 server on CentOS 4 that front-ends a couple of App servers (mix of Jetty and Tomcat). Apache has a handful of virtual hosts configured like this:

<VirtualHost www1.example.com:443>
    ServerName www1.example.com
    DocumentRoot "/mnt/app_web/html"

    SSLEngine on
    SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP
    SSLCertificateFile      /etc/httpd/conf/ssl.crt/server.crt
    SSLCertificateChainFile /etc/httpd/conf/ssl.crt/chain.crt
    SSLCertificateKeyFile   /etc/httpd/conf/ssl.key/server.key
    SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0

    RewriteEngine on
    RewriteRule ^/app1/(.*)$ http://app1.example.com:8080/app1/$1 [P,L]
    RewriteRule ^/app2/(.*)$ http://app2.example.com:8080/app2/$1 [P,L]
</VirtualHost>

However, I'm getting the following errors in the logs intermittently:

[Fri Dec 04 07:19:41 2009] [error] (113)No route to host: proxy: HTTP: attempt to connect to 10.0.0.1:8080 (app1.example.com) failed

I initially tried turning off IPv6, and that seemed to largely cure it, but I still have sporadic bursts of these messages.

Additionally, we're running memcache on same front-end and during the times when I'm getting those messages in Apache's log, the following command doesn't work:

echo stats | nc 127.0.0.1 11211

No messages are printed, but neither are the stats printed. I am completely lost as to how to proceed with troubleshooting this. =(

Solution 1

To solve this problem you need to add rule(s) in the 'iptables' of your App servers. For Red Hat Enterprise the file is '/etc/sysconfig/iptables' . It should be the same for CentOS.

You probaly have one or more rules that accept NEW connection from the front-ends that look like these:

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s 'IP of the front-ends' --dport 'port number' -j ACCEPT

OR

-A RH-Firewall-1-INPUT -m state --state NEW -m multiport -m tcp -p tcp -s 'IP of the front-ends' --dports 'ports numbers' -j ACCEPT

Your problem shoud be solved by adding rules that send a tcp-reset to the front-ends for each SYN packet that passed throught the preceedings rules. The rules should looks like these:

-A RH-Firewall-1-INPUT -m tcp -p tcp -s 'IP of the front-ends' --dport 'port number' --syn -j REJECT --reject-with tcp-reset

OR

-A RH-Firewall-1-INPUT -m multiport -m tcp -p tcp -s 'IP of the front-ends' --dports 'ports numbers' --syn -j REJECT --reject-with tcp-reset

Add the rules near the end of your 'iptables' just before the rule that looks like:

-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited

Good luck.

Paul

Solution 2

I had this problem (113 - No route to host) on CentOS 6.5 with Apache 2.2, although it was only intermittent- about every 20 mins. I doubt it is related to your issue but this may help someone.

I captured the network traffic on both ends using Wireshark and discovered that ICMP host administratively prohibited packets were sometimes being returned by the App server in response to a SYN packet from the Reverse Proxy. However, most of the time the SYNs were being accepted.

Based on Paul's answer, I checked the iptables rules- indeed the rules of course were fine to accept NEW connections on port 8009.

I decided to reset iptables with the command:

service iptables restart

After restarting iptables the '113 - No route to host' problem has gone away completely.

Comments

[BonkaBonka]

While this didn't exactly fix the problem, it did point the way to a proper resolution - specifically the iptables rules allowed access to tomcat only if the request was IPv4 (d'oh!) We also shuffled a couple of PHP apps off the front-end server to another VM and we've been error-free for a couple of months now. Thanks! (I'd upboat but I don't have the rep on this shard =/)

[BonkaBonka]

This came back to bite us again a while ago and we finally found the root cause - IPTables connection tracking. Turning that off truly cured the problem.

[BonkaBonka]

IPv6 was a red-herring. The root cause is IPTables' connection tracking. Disabling it has permanently cured the issue.