Varnish automagically adding load balancer IP to X-Forwarded-For header

Question

My request flow is as follows;

HAProxy --> Varnish (4.0.1) --> Apache web backends

When a new request comes in to HAProxy, the client's IP address is being added to the X-Forwarded-For header (which is good!). However, it looks like Varnish is adding the HAProxy IP as well. When the request gets to my vcl_recv routine, the X-Forwarded-For header is:

X-Forwarded-For: end-user-ip, haproxy-ip

You can see that in the varnishlog output:

*   << Request  >> 8
-   Begin          req 7 rxreq
-   Timestamp      Start: 1409262358.542659 0.000000 0.000000
-   Timestamp      Req: 1409262358.542659 0.000000 0.000000
-   ReqStart       192.168.1.103 48193
-   ReqMethod      PURGE
-   ReqURL         /some/path
-   ReqProtocol    HTTP/1.1
-   ReqHeader      Authorization: Basic xxx
-   ReqHeader      User-Agent: curl/7.30.0
-   ReqHeader      Host: example.com
-   ReqHeader      Accept: */*
-   ReqHeader      X-Forwarded-For: 1.2.3.4
-   ReqHeader      Connection: close
-   ReqUnset       X-Forwarded-For: 1.2.3.4
-   ReqHeader      X-Forwarded-For: 1.2.3.4, 192.168.1.101
-   VCL_call       RECV
-   ReqUnset       X-Forwarded-For: 1.2.3.4, 192.168.1.101
-   VCL_acl        NO_MATCH purge_acl
-   Debug          "VCL_error(403, Not allowed.)"
-   VCL_return     synth

The reason I need the accurate client IP address is so I can check it against ACL rules for PURGE/BAN. Since the last IP in the X-Forwarded-For header is that of HAProxy, the ACL check fails for all IPs. Here is the relevant section of my config:

acl purge_acl {
    "1.2.3.4";
}

sub vcl_recv {

    set req.backend_hint = load_balancer.backend();

    if (req.method == "PURGE") {
        if (!std.ip(req.http.X-forwarded-for, "0.0.0.0") ~ purge_acl) {
            return(synth(403, "Not allowed."));
        }
        ban("obj.http.x-url ~ " + req.url);
        return(synth(200, "Ban added"));
    }

}

Any ideas how I can rely solely on the X-Forwarded-For header from HAProxy, without Varnish tampering with it?

A side note, it seems that Varnish is doing exactly this (although this IS NOT in mv VCL config):

if (req.restarts == 0) {
    if (req.http.X-Forwarded-For) {
        set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
    } else {
        set req.http.X-Forwarded-For = client.ip;
    }
}

Solution 1

I also bumped into this problem today.

The default.vcl in varnish 4.0 was renamed in builtin.vcl and does not contain the set req.http.X-Forwarded-For part that you mentioned - link. Nevertheless he clearly appends to a comma separated list the intermediate proxy IP address as per the protocol specs - Wikipedia link.

One solution would be to use the X-Real-IP header instead, overwriting this header all the time in HAProxy with the real client ip and using this one for vcl ACL.

Another solution as (wrongly) mentioned in the varnish forum, would be regsub(req.http.X-Forwarded-For, "[, ].*$", "") that takes the leftmost IP address. However, this method is NOT SECURE, since this header can be easily spoofed.

My suggestion would be to extract the known part, varnish IP from the header like this:

if (!std.ip(regsub(req.http.X-Forwarded-For, ", 192\.168\.1\.101$", ""), "0.0.0.0") ~ purge_acl) {
    return(synth(403, "Not allowed."));
}

The only problem with this is if there are more than 2 hops in the connection, eg. you also use a proxy to connect to internet. A good solution for this is provided by nginx since you can define trusted hops, and they are ignored recursively until the real client ip.

set_real_ip_from 192.168.1.101;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

You can see more details about this in this serverfault thread answer

You might also want to check why in your VCL_call RECV you do a ReqUnset X-Forwarded-For BEFORE the ACL match.

Solution 2

Varnish appends its default logic to any functions you define such as vcl_recv rather than purely overriding it. The default vcl_recv logic contains:

if (req.http.x-forwarded-for) {
    set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
} else {
    set req.http.X-Forwarded-For = client.ip;
}

as you noticed. What seems strange to me is that it seems like the Varnish default logic in vcl_recv is executing ahead of your vcl_recv logic. What value do you see for X-Forwarded-For within vcl_deliver if you define your own?

One thing you could do is parse out the first IP address like this where necessary:

set req.http.X-Forwarded-For = regsub(req.http.X-Forwarded-For, "^([^,]+),?.*$", "\1");

Solution 3

The Vanish source code has moved to GitHub so, for reference, since version 4.0 the X-Forwarded-For logic has been moved out of the builtin.vcl, (formerly default.vcl), and the source logic can be found in bin/varnishd/cache/cache_req_fsm.c.

Comments

[Josh]

This answer helped me the most.

[Cosimo]

This answer does not hold for Varnish 4.x and upwards. See stackoverflow.com/a/40840890/11303