Apache log showing NAT server address instead of real requester IP

Question

My Apache server is behind a firewall with a NAT translation. The problem I'm having is that I want to see who is actually making the request instead of my firewall's address. It's nice to know that the firewall is working, but to really look at traffic patterns I need to see the real world IP address.

UPDATE

Firewall is a CentOS 5.2 box using iptable rules created by fwbuilder.

iptables reponds to requests on all interfaces, Squid is running on internal facing interfaces only.

Solution 1

I've seen this with proxy servers and load balancers. The usual case is that inbound traffic crosses the proxy to get to the Web server, but the Web server's default gateway is something other than the proxy. By doing reverse NAT, the Web server gets the proxy's IP instead of the client's. Since it will have a route to the proxy (and in fact is probably in the same subnet), is assured that it can always get return traffic back to the client.

One fix for this is for the proxy to insert a custom HTTP header containing the client's real IP into the HTTP request that the Web server can parse. For Apache, it becomes a simple problem of modifying your LogFormat statement to use %{Custom-Header} instead of %h. Of course, this depends on your device actually being HTTP aware and being capable of inserting arbitrary headers into GET/POST/etc. requests. It's a common feature for proxies and load balancers, but not so much for firewalls. Additionally, unless your device is doing SSL termination, it won't help you for HTTPS requests. As Kyle said, we need to know more about your firewall.

Solution 2

I think most common with NAT in this case it to only NAT the destination address. Therefore the clients will still have the same source address (the public ip). So I think we need to know more about the firewall.

So for example:

Client: 12.12.12.12
Public Website Address: 11.11.11.11
Private Web Server ip: 10.0.0.2

Client -->  Your Firewall --> Web Server

Client Packet:
    Src IP:12.12.12.12
    Dst IP:11.11.11.11
NAT Firewall:
    Takes client Packet, changes Dst from 11.11.11.11 to 10.0.0.2.
    Make entry in table to remember this mapping
Web Server (When Client Packet Arrives):
    Src:12.12.12.12
    Dst: 10.0.0.2

Reply Packet from Web Server:
    Src: 10.0.0.2
    Dst: 12.12.12.12
NAT Firewall:
    Takes Reply Packet, changes Src to 11.11.11.11 after looking at mapping
Client (When Packet Arives):
    Src: 11.11.11.11 
    Dst: 12.12.12.12

Chances are the client is also behind NAT, but that just makes it more confusing for this purpose.

Solution 3

From how you're describing it, this "firewall" looks a lot more like a reverse proxy.

NAT is usually applied to outgoing connections, not to incoming ones. When a computer in the private network opens a connection to the outside world, the remote server sees the connection as coming from the firewall's public IP address; but when a remote computer opens a connection to one of your internal servers (through a port forwarding on the firewall), it sees the connection as coming from the actual public IP address it's originating from. What you're describing is the typical behaviour of a reverse proxy, not of a NAT firewall.

Are you sure there's not any reverse proxy (like SQUID) running on that firewall and intercepting incoming HTTP(S) connections? SQUID can also act as a transparent proxy, so it could be there without you having any knowledge of that. This would be also the standard behaviour of ISA Server when you publish an internal web site: even if you think you're just doing a port forwarding, you're actually reverse-proxying your web server.

Solution 4

If you have squid in the mix, you should look for the X-Forwarded-For header which was implemented specifically for the purpose of passing that information forward.

Note that this header is at least as insecure as using the origin ip as both can be faked.

You can test if this is the case by modifying your LogFormat replacing %h with %{X-Forwarded-For}i and then reload your configuration.

View more solutions

Comments

[Admin]

Posting the internal network configuration and the IPTABLES rules (masking the public IP addresses if you feel so) would help a lot.

[Admin]

Double check what you are doing with fwbuilder, because masquerading source IP on lan side is not so useful.

[Admin]

I don't have direct access to the fwbuilder script output so I can't say exactly what our iptable rules are. That being said, there is a series of public ip addresses that map their respective public ports (80, 443 in most cases) to the internal server. Internally our firewall's address is 10.255.255.3 and all of the webservers it points to are servers in the 10.0.1.x range. We have two outbound squid proxies and all incoming traffic comes in on 10.255.255.3, which also houses one of the squid proxies.

[Admin]

If you have root access, you can see the resulting rules with iptables -L nat -n -v.

[Kyle Brandt]

+1 Ah reverse proxy, good call.

[thaBadDawg]

I'm using iptables on a CentOS box. The iptable rules were created by fwbuilder.

[thaBadDawg]

Squid is running on the box but it doesn't attach to the public network interface. iptables is the only thing running that connects to both interfaces. (Actually, that's a lie, because SSH also responds on the public interface.)

[Massimo]

Try opening a telnet connection to the external IP address on port 80 and see who is actually answering you. When it's open, use "netstat" on both boxes to see who is actually handling the connection.