Haproxy not properly passing on X-Forwarded-For header

Question

I have backend web servers that receive requests by way of haproxy->nginx->fastcgi. The web app used to see multiple ip's coming through in the X-Forwarded-For header, chained together with commas (most original IP on the left).

At some point in the recent past (just noticed, so not sure what caused it) something changed, and now I'm only seeing a single IP passed in the header to my web application.

I've tried with haproxy 1.4.21 and 1.4.22 (recent upgrade) with the same behavior. Haproxy has the forwardfor header set:

option forwardfor

Nginx fastcgi_params config defines this header to be passed to the app:

fastcgi_param HTTP_X_FORWARDED_FOR $http_x_forwarded_for;

Anyone have any ideas on what might be going wrong here?

EDIT: I just started logging the $http_x_forwarded_for variable in nginx logs, and nginx is only ever seeing a single IP, which shouldn't ever be the case, as we should always see our haproxy ip added in there, right? So, issue must either be in nginx handling of the variable coming in, or haproxy not building it properly. I'll keep digging...

EDIT #2: I enabled request and response header logging in HAProxy, and it is not spitting anything out for X-Forwarded-For, which seems very odd:

Oct 10 10:49:01 newark-lb1 haproxy[19989]: 66.87.95.74:47497 [10/Oct/2012:10:49:01.467] http service/newark2 0/0/0/16/40 301 574 - - ---- 4/4/3/0/0 0/0 {} {} "GET /2zi HTTP/1.1" O

Here are the options i set for this in my frontend:

mode http
option httplog
capture request header X-Forwarded-For len 25
capture response header X-Forwarded-For len 25
option httpclose
option forwardfor

EDIT #3: It really seems like haproxy is munging the header and just passing on a single one to the backend. This is fairly impacting to our production service, so if anyone has an ideas it would be greatly appreciated. I'm stumped... :(

Solution

To respond to your last question in the comment, it is normal to have more than one IP address in XFF, this header is a list of values, and proxies often add their client's address there. Since everyone in the long chain appends values there, your server must use them in reverse order. For instance, the last value will be the one added by the haproxy instance in front of the server, and the previous value will be the one added by the reverse-cache before haproxy, etc...

If you'd prefer not to adapt the application to correctly parse the header, you can also ask haproxy to remove it before adding its own XFF header:

reqidel ^X-Forwarded-For:

That way the server will only get the value added by haproxy which will be haproxy's client.

Comments

[JesseP]

Willy, thanks for the response. while I did have something in front of haproxy at one point, I am pretty sure that haproxy frequently sent on XFF headers with a chain of IPs in the past. Did this used to happen in older versions, where the newer versions just send on the just one IP? At some point i wrote code in my app to pull the proper IP from the XFF chain, instead of the request IP, if the XFF header was present (at least my haproxy in the request path) so i didn't geo-target the user to the location of my load balancer.

[JesseP]

Is there a better way (at least with >= haproxy 1.4.21) to determine the clients IP for geolocation purposes (not their private IP incase they are running through a proxy)?

[JesseP]

Also,thanks for the suggestion around keep alive. I'll be sure to give that a shot for our website and client dashboard to see how it improves load times. :). Or service,where we use geoip most, is usually just single requests, so we chose to disable keep alive there.

[Willy Tarreau]

I don't get your point then. The client's IP is sent in the XFF header to the server. Concerning keep-alive, you may then use "forceclose" to disable it more efficiently than with "httpclose" as it releases the server connection earlier.

[JesseP]

Is there a way to put the clients public ip in the header instead of their private, if they are passing through a proxy on the way? I see a good handful of 10., 192., etc (private ips) in the header the today, which I'm of course unable to geo-locate.

[JesseP]

Also, just to clarify you say: "The client's IP is sent in the XFF header to the server" Are you saying that HAProxy intentionally only adds a single IP to the XFF header, for the clients actual IP address? (even if there were already one or more XFF IP addresses in the header) If this is the case, what I want to be able to extract on my server side is the first public IP in the chain from the client, so I can use it for GeoIP resolution. Is there a way to do this? My current implementation sometimes returns a private ip.

[JesseP]

Willy, the problem is that I used to ser multiple addresses in the header and wrote code in my application to pull out the first non-private IP in the chain and use it, which worked great. A some point something changed and now I only ever receive a single address in the header at my application level, which seems like haproxy is applying that reqidel logic by default or something. Do you see any reason that would be overwriting the chain instead of appending, based on my config?

[Willy Tarreau]

It does not do that by default, and I suspect that your application does not iterate over all headers but randomly picks one and uses the value it carries. Haproxy never modifies the existing header, it always adds one with the new value. It may be the one your application is seeing, but your application should also see the previous ones.