DNS lookup failures while accessing my website some proxy error

Question

Here is a situation until today morning,every thing has been working perfectly fine with me. From past 6 months many of my domains wer accessible as

http://site1.myserver.com
http://site2.myserver.com
http://site3.myserver.com
http://site4.myserver.com

All these were Reverse Proxy configurations. I have some applications on each of them.

until today morning some people reported me that
http://site1.myserver.com/app1 is not working but http://site1.myserver.com is accessible
but http://site2.myserver.com is accessible
but http://site3.myserver.com is accessible
but http://site4.myserver.com not accessible

In past 6 months I have not changed any of these Apache configurations
(things were working perfectly so)

The error which can be seen in browser are while accessing http://site1.myserver.com/app1

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /app1.

Reason: DNS lookup failure for: myserver.com

and same is the error for http://site4.myserver.com

So what should I check in I have checked all the apache logs to an extent which I could see and

    192.168.1.25 - - [10/Jan/2011:14:50:48 +0530] "GET /app1 HTTP/1.1" 502 531 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"

Mon Jan 10 14:27:42 2011] [error] (113)No route to host: proxy: HTTP: attempt to connect to 192.168.1.3:80 (192.168.1.3) failed
[Mon Jan 10 14:27:42 2011] [error] ap_proxy_connect_backend disabling worker for (192.168.1.3)
[Mon Jan 10 14:27:44 2011] [error] proxy: HTTP: disabled connection for (192.168.1.3)
[Mon Jan 10 14:27:44 2011] [error] proxy: HTTP: disabled connection for (192.168.1.3)
[Mon Jan 10 14:27:44 2011] [error] proxy: HTTP: disabled connection for (192.168.1.3)
[Mon Jan 10 14:27:45 2011] [error] proxy: HTTP: disabled connection for (192.168.1.3)
[Mon Jan 10 14:27:45 2011] [error] proxy: HTTP: disabled connection for (192.168.1.3)
[Mon Jan 10 14:27:45 2011] [error] proxy: HTTP: disabled connection for (192.168.1.3)
[Mon Jan 10 14:27:45 2011] [error] proxy: HTTP: disabled connection for (192.168.1.3)
[Mon Jan 10 14:27:46 2011] [error] proxy: HTTP: disabled connection for (192.168.1.3)
[Mon Jan 10 14:27:47 2011] [error] proxy: HTTP: disabled connection for (192.168.1.3)
[Mon Jan 10 14:27:48 2011] [error] proxy: HTTP: disabled connection for (192.168.1.3)
[Mon Jan 10 14:27:48 2011] [error] proxy: HTTP: disabled connection for (192.168.1.3)
[Mon Jan 10 14:27:48 2011] [error] proxy: HTTP: disabled connection for (192.168.1.3)
[Mon Jan 10 14:35:29 2011] [error] [client 192.168.1.25] proxy: DNS lookup failure for: myserver.com returned by /app1
[Mon Jan 10 14:35:30 2011] [error] [client 192.168.1.25] proxy: DNS lookup failure for: myserver.com returned by /app1
[Mon Jan 10 14:35:30 2011] [error] [client 192.168.1.25] proxy: DNS lookup failure for: myserver.com returned by /app1
[Mon Jan 10 14:50:30 2011] [error] [client 192.168.1.25] proxy: DNS lookup failure for: myserver.com returned by /app1
[Mon Jan 10 14:50:48 2011] [error] [client 192.168.1.25] proxy: DNS lookup failure for: myserver.com returned by /app1

and for site4.myserver.com

I get

[Mon Jan 10 14:57:40 2011] [error] [client 192.168.1.25] proxy: DNS lookup failure for: site4.myserver.com returned by /favicon.ico
[Mon Jan 10 14:57:40 2011] [error] [client 192.168.1.25] proxy: DNS lookup failure for: site4.myserver.com returned by /favicon.ico
[Mon Jan 10 14:57:43 2011] [error] [client 192.168.1.25] proxy: DNS lookup failure for: site4.myserver.com returned by /favicon.ico
[Mon Jan 10 15:02:38 2011] [error] [client <some external IP>] proxy: DNS lookup failure for: site4.myserver.com returned by /
[Mon Jan 10 15:03:04 2011] [error] [client <some external IP>] proxy: DNS lookup failure for: site4.myserver.com returned by /, referer: http://site4.myserver.com/
[Mon Jan 10 15:03:04 2011] [error] [client <some external IP>] proxy: DNS lookup failure for: site4.myserver.com returned by /favicon.ico
[Mon Jan 10 15:03:08 2011] [error] [client <some external IP>] proxy: DNS lookup failure for: site4.myserver.com returned by /, referer: http://site4.myserver.com/
[Mon Jan 10 15:03:08 2011] [error] [client <some external IP>] proxy: DNS lookup failure for: site4.myserver.com returned by /favicon.ico
[Mon Jan 10 15:03:10 2011] [error] [client <some external IP>] proxy: DNS lookup failure for: site4.myserver.com returned by /, referer: http://site4.myserver.com/
[Mon Jan 10 15:06:21 2011] [error] [client 192.168.1.25] proxy: DNS lookup failure for: site4.myserver.com returned by /
[Mon Jan 10 15:06:31 2011] [error] [client 192.168.1.25] proxy: DNS lookup failure for: site4.myserver.com returned by /, referer: http://site4.myserver.com/
[Mon Jan 10 15:26:03 2011] [error] [client 192.168.1.25] proxy: DNS lookup failure for: site4.myserver.com returned by /

Solution 1

This problem was solved by installing prodns and dnsproxy.

Solution 2

If YOU haven't changed anything, then something else has obviously.

You need to check DNS resolution from your reverse proxy server.

It sounds like you may have a cached, negative response on the DNS server your proxy is pointed to.

Comments

[Mike Scott]

We can't check your DNS unless you tell us the actual domain name.

[Bond]

The problem is I do not have the control of DNS it is some third person who has its control (which is not in my hands).So I can not check any thing on DNS.How ever I have given link to URL as above so you can check them.

[Essobi]

You DO have the ability to perform lookups on the reverse proxy server? your server appears to be failing lookups... Either run your own DNS server, or add them directly to the hostfile on the reverse proxy server, and you shouldn't perform lookups on them. I would only do that if the servers don't plan on changing IP addresses very often.

[Bond]

I found the exact cause of problem after having a long trouble shooting hours found that my proxy pass entries if look as follows ProxyPass /app1 192.168.1.3:8080/app1 ProxyPass / 192.168.1.3 ProxyPassReverse /app1 192.168.1.3:8080/app1 ProxyPassReverse / 192.168.1.3 then some one from internet is able to access the sites.But from within LAN people are unable to access it. How ever if I use

[Bond]

ProxyPass /app1 site1.myserver.com:8080/app1 ProxyPass / site1.myserver.com ProxyPassReverse /app1 site1.myserver.com:8080/app1 ProxyPassReverse / site1.myserver.com then things work on LAN but from internet it is not accessible and the error I get is the one I mentioned as above. What can be a solution in this case will putting things in resolv.conf do?

[Bond]

Or I need some thing else to do?

[Essobi]

I'm unsure what your proxy is doing exactly. Hardcoded IP addresses should be defined in the 'hosts' file. Using this overrides the servers ability to lookup those hosts. If they change you'll have to redefine them by hand. If it were me, I'd get the external working, and setup a DNS caching server that can override host domains, and point those to the internal IPs, and set all of the LAN clients to use that DNS server to override the IPs. Either way, you likely won't be able to access them from the WAN and LAN at the same time in this setup.

[Bond]

I agree with you to debug this I used tcpdump to analyse the virtual machines (different from Apache vhosts) and found they were sending requests to port 53 for DNS lookups to corporate DNS which did not had the entries.Though the internal entries were there in /etc/hosts even then the Virtual Machines were sending requests to corporate DNS. I installed prodns and dnsproxy as per your suggestion for serving these DNS lookups since corporate DNS will not have entry to these machines internally.This basically served as cachced DNS and then things are working very well.Thanks for your help.

[Essobi]

You're welcome. Uprate me or something for answering your question thou... Actually, I think you're supposed to accept my answer for helping you, not write your own and check it. :\