nginx real_ip_header and X-Forwarded-For seems wrong

Question

The wikipedia description of the HTTP header X-Forwarded-For is:

X-Forwarded-For: client1, proxy1, proxy2, ...

The nginx documentation for the directive real_ip_header reads, in part:

This directive sets the name of the header used for transferring the replacement IP address.
In case of X-Forwarded-For, this module uses the last ip in the X-Forwarded-For header for replacement. [Emphasis mine]

These two descriptions seem at odds with one another. In our scenario, the X-Forwarded-For header is exactly as described -- the client's "real" IP address is the left-most entry. Likewise, the behavior of nginx is to use the right-most value -- which, obviously, is just one of our proxy servers.

My understanding of X-Real-IP is that it is supposed to be used to determine the actual client IP address -- not the proxy. Am I missing something, or is this a bug in nginx?

And, beyond that, does anyone have any suggestions for how to make the X-Real-IP header display the left-most value, as indicated by the definition of X-Forwarded-For?

Solution 1

I believe the key to solving X-Forwarded-For woes when multiple IPs are chained is the recently introduced configuration option, real_ip_recursive (added in nginx 1.2.1 and 1.3.0). From the nginx realip docs:

If recursive search is enabled, an original client address that matches one of the trusted addresses is replaced by the last non-trusted address sent in the request header field.

nginx was grabbing the last IP address in the chain by default because that was the only one that was assumed to be trusted. But with the new real_ip_recursive enabled and with multiple set_real_ip_from options, you can define multiple trusted proxies and it will fetch the last non-trusted IP.

For example, with this config:

set_real_ip_from 127.0.0.1;
set_real_ip_from 192.168.2.1;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

And an X-Forwarded-For header resulting in:

X-Forwarded-For: 123.123.123.123, 192.168.2.1, 127.0.0.1

nginx will now pick out 123.123.123.123 as the client's IP address.

As for why nginx doesn't just pick the left-most IP address and requires you to explicitly define trusted proxies, it's to prevent easy IP spoofing.

Let's say a client's real IP address is 123.123.123.123. Let's also say the client is up to no good, and they're trying to spoof their IP address to be 11.11.11.11. They send a request to the server with this header already in place:

X-Forwarded-For: 11.11.11.11

Since reverse proxies simply add IPs to this X-Forwarded-For chain, let's say it ends up looking like this when nginx gets to it:

X-Forwarded-For: 11.11.11.11, 123.123.123.123, 192.168.2.1, 127.0.0.1

If you simply grabbed the left-most address, that would allow the client to easily spoof their IP address. But with the above example nginx config, nginx will only trust the last two addresses as proxies. This means nginx will correctly pick 123.123.123.123 as the IP address, despite that spoofed IP actually being the left-most.

Solution 2

The parsing of the X-Forwarded-For header is indeed flawed in the nginx real_ip module.

len = r->headers_in.x_forwarded_for->value.len;
ip = r->headers_in.x_forwarded_for->value.data;

for (p = ip + len - 1; p > ip; p--) {
  if (*p == ' ' || *p == ',') {
    p++;
    len -= p - ip;
    ip = p;
    break;
  }
}

It starts on the far right of the header string, and as soon as it sees a space or comma, it stops looking and sticks the part to the right of the space or comma in the IP variable. So, it's treating the most recent proxy address as the original client address.

It's not playing nice according to the spec; this is the danger of not having it spelled out in painfully obvious terms in an RFC.

Aside: It's hard to even find a good primary source on the format, which was originally defined by Squid - a dig through their documentation confirms the ordering; leftmost is original client, rightmost is the most recent append. I'm sorely tempted to add a [citation needed] to that wikipedia page. One anonymous edit seems to be the internet's authority on the subject.

If possible, can you have your intermediate proxies stop adding themselves to the end of the header, just leaving it with the real client address only?

Solution 3

X-Real-IP is the IP address of the actual client the server is talking to (the "real" client of the server), which, in the case of a proxied connection, is the proxy server. That's why X-Real-IP will contain the last IP in the X-Forwarded-For header.

Comments

[Kirk Woll]

OK, but, for me, that is simply never useful information. I want to get the original IP address of the client -- that is crucial, and according to everything I've read, the purpose of these headers. Why would I want to know the IP address of our proxy servers?

[user558061]

If it's not useful for you then it's not for you. Nobody's forcing you to use X-Real-IP. If you need the IP of the user in your application, then have your application parse X-Forwarded-For (which isn't always reliable because there are some proxies (internet security appliance/firewalls) that don't set X-Forwarded-For). In the context of nginx, X-Forwarded-For is not important because it's not talking to those clients anyway, aside from the last entry (the X-Real-IP) which is nginx's client. If you don't need it then don't set it, unset it, or just ignore it :/

[Kirk Woll]

No, what I mean is, why would X-Real-IP returning my own proxy server's IP address ever be useful?

[Kirk Woll]

Thanks for the reply, @Shane. In fact, when reaching nginx, an X-Forwarded-For already exists. (it is the correct client IP address) nginx itself then proceeds to append the IP address of our load balancer (the previous hop) to the X-Forwarded-For header. (presumably appending what it sees as the "remote address") If it simply did not do that, I would be able to just use the X-Forwarded-For header as before. (we've recently been migrating to nginx)

[ravi yarlagadda]

@Kirk So, when nginx gets the header, it's just the original client's address? But when it's processing it, it's added on the connecting proxy server's header? That doesn't add up - the only time it should touch that header is when it's sending the connection to another proxy via a proxy_pass - and even then, only with proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; in place.

[Yugal Jindle]

Great.. answer man. I was looking for this precise information. I need to talk to a ncat server on my proxy server, so I need this on the fly.

[gansbrest]

By default real_ip_header seem to be X-Real-IP according to nginx.org/en/docs/http/ngx_http_realip_module.html Does it mean malicious user can just send request with random X-Real-IP and that will be used as $remote_addr in nginx (and also possibly passed to the application)?

[Ian Kemp]

Even the W3C gets this wrong: their documentation states "proxies should add the IP address of the initiator of the request to the end of a comma separated list in an X-Forwarded-For HTTP header field", it should state beginning.

[blubberdiblub]

@IanKemp, no, end is correct. To the server side of a proxy, the initiator of the request (i. e. TCP request) is the previous proxy (if there is one). That previous proxy possibly already sends an X-Forwarded-For header with possibly the original client address at the left and possibly any preceding proxies appended to that. So the currently serving proxy would add the previous proxy (= initiator) to the end of that list and serve the thus augmented X-Forwarded-For header to the next upstream hop. Granted, they could have chosen a more obvious wording.

[El Yobo]

@gansbrest No, because set_real_ip_from limits the trusted hosts.

[Mert Z.]

You also need to include the $remote_addr (IP of the proxy before nginx which under normal circumstances is not included in X-Forwarded-For header) in the set_real_ip_from directive along with the other proxies in X-Forwarded-For header.

[THE JOATMON]

I know I'm a decade late but this still isn't working for me. Everything shows up as 10.1.0.1, the address of my firewall/gateway, if it's an internal source. External sources show up properly.

[Admin]

Surprised this issue dates back to 2011... Personally I think real_ip_recursive should be ON by default - the whole purpose of the set_real_ip_from feature is to know the origin ip, not ip to a proxied server.