Nginx TCP forwarding based on hostname

nginx tcp load-balancing virtualhost nginx-reverse-proxy

71,277

Solution 1

Assumptions

If I understand you correctly, you effectively want nginx to listen at a single IP address and TCP port combination (e.g., listen 10.0.0.1:443), and then, depending on the characteristic of the incoming TCP stream traffic, route it to one of the 3 different IP addresses.

You don't explicitly mention how you expect it to differentiate between the 3 different domains at stake, but my assumption is that you assume it's all just TLS, and must want to employ some sort of a TLS SNI (Server Name Indication) mechanism for domain-based differentiation.

I would believe that the stream-related documentation provided at http://nginx.org/docs/ is quite authoritative and exhaustive for the modules at stake (I'm listing all of it here, since apparently there's no central place for cross-referencing this yet, e.g., no references from the "stream core" module to the submodules yet (and docs/stream/ just redirects back docs/), which is indeed quite confusing, since stuff like http://nginx.org/r/upstream is only documented to apply to http, without any mention of applicability to stream, even if the directives are about the same in the end):

Answer

Note that each nginx directive, from each module, has a limited number of applicable Context's.

As such, unfortunately, there is simply no directive to snoop into SNI here!

To the contrary, it's actually documented in stream_core that, to quote, "Different servers must listen on different address:port pairs.", which, as you may note, is also contrary to how the listen directive works within the more-common http_core, and is a rather unambiguous reference to the fact that no kind of SNI support is presently implemented for the listen within stream.

Discussion

As a discussion point and a resolution suggestion, the assumption that OpenVPN traffic is just TLS with the snoopable SNI is also not necessarily correct (but I'm not too familiar with OpenSSL or SNI):

Consider that even if SNI is passively snoopable today, that's clearly contrary to the promise of TLS of keeping the connection secure, and, as such, may change in a future version of TLS.
For the sake of discussion, if OpenVPN is just using a TLS connection, and if it is NOT using TLS for authenticating users with user certificates (which would make it much more difficult to MitM the stream, yet still carry the authentication data all along), then, theoretically, if nginx did have SNI support around the listen within stream, then you'd possibly have been able to actively MitM it with nginx (since proxy_ssl is already supported in stream_proxy).

Most importantly, I believe OpenVPN may best be run over its own UDP-based protocol, in which case, you can use the same IP address and port number for one instance of the TCP-based https and another one of the UDP-based OpenVPN without a conflict.

In the end, you may ask, what would the stream module be useful for anyways, then? I believe its target audience would be, (0), load balancing HTTP/2 with multiple upstream servers, based on the hash of the IP-address of the client, for example, and/or, (1), a more straightforward and protocol-agnostic replacement for stunnel.

Solution 2

This is now possible with the addition of the ngx_stream_ssl_preread module added in Nginx 1.11.5 and the ngx_stream_map module added in 1.11.2.

This allows Nginx to read the TLS Client Hello and decide based on the SNI extension which backend to use.

stream {

    map $ssl_preread_server_name $name {
        vpn1.app.com vpn1_backend;
        vpn2.app.com vpn2_backend;
        https.app.com https_backend;
        default https_default_backend;
    }

    upstream vpn1_backend {
        server 10.0.0.3:443;
    }

    upstream vpn2_backend {
        server 10.0.0.4:443;
    }

    upstream https_backend {
        server 10.0.0.5:443;
    }

    upstream https_default_backend {
        server 127.0.0.1:443;
    }

    server {
        listen 10.0.0.1:443;
        proxy_pass $name;
        ssl_preread on;
    }
}

Solution 3

AS @Lochnair mentioned, you can use ngx_stream_map module and variable $server_addr to resolve this problem. Here is my example.

My host IP is 192.168.168.22, and I use keepalived bound 2 virtual IP to eth0.

$sudo ip a
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 5c:f3:fc:b9:f0:84 brd ff:ff:ff:ff:ff:ff
inet 192.168.168.22/24 brd 192.168.168.255 scope global eth0
   valid_lft forever preferred_lft forever
inet 192.168.168.238/32 scope global eth0
   valid_lft forever preferred_lft forever
inet 192.168.168.239/32 scope global eth0
   valid_lft forever preferred_lft forever

$nginx -v
nginx version: nginx/1.13.2

$cat /etc/nginx/nginx.conf
...
stream {
    upstream pod53{
        server 10.1.5.3:3306;
    }
    upstream pod54{
        server 10.1.5.4:3306;
    }

    map $server_addr $x {
        192.168.168.238 pod53;
        192.168.168.239 pod54;
    }
    server {
        listen 3306;
        proxy_pass $x;
    }
}

Thus, I can visit different MySQL service with the same port 3306 via different VIPs. Just like visiting different HTTP service with the same port via diffrent server_name.

192.168.168.238 -> 10.1.5.3
192.168.168.239 -> 10.1.5.4

71,277

Author by

James Wong

From Mauritius. James supports Monica Cellio. Read more about this on "Firing mods and forced relicensing: is Stack Exchange still interested in cooperating with the community?"

Updated on July 05, 2022

Comments

James Wong almost 2 years
With the release of TCP load balancing for the Nginx community version, I would like to mix OpenVPN and SSL pass-through data. The only way for Nginx to know how to route the traffic is via their domain name.
```
 vpn1.app.com ─┬─► nginx at 10.0.0.1 ─┬─► vpn1  at 10.0.0.3
 vpn2.app.com ─┤                      ├─► vpn2  at 10.0.0.4
https.app.com ─┘                      └─► https at 10.0.0.5
```
I have taken a look at the TCP guides and the module documentation, but it doesn't seem well referenced. If anyone can point me to the right direction, i'd be grateful.

Related question on ServerFault: Can a Reverse Proxy use SNI with SSL pass through?
cnst over 7 years

@TorstenBronger, I don't think people read the body of the question, which specifically deals with OpenVPN. Does OpenVPN itself even support SNI?
cnst about 7 years

@TorstenBronger, BTW, it so turns out that OpenSSL does not support SNI, so, the question is moot, and thus the other answer is answering a different question. Source: stackoverflow.com/questions/42078600/…
Torsten Bronger about 7 years

Once nginx has routed by hostname, OpenVPN doesn't need SNI anymore.
Dae about 7 years

To whoever arrives here in the future thinking this could work for SSH connections - don't bother, SSH clients don't pass the DNS name.
cnst over 6 years

@Dae, SSH is not even related to SSL.
Lapsio over 6 years

in such case if I set up conventional https proxy listening on 127.0.0.1:443 I can chain stream with normal http proxy in nginx? For example to use nginx as vpn/httos reverse proxy but also as SSL offload proxy for https.
Lochnair over 6 years

@Lapsio I'm a bit unclear on exactly what setup you're trying to achieve, but yes having a TCP stream in front of a HTTP web server works perfectly fine.
Lapsio over 6 years

@Lochnair yeah i did it, hosting stream and http proxy on one machine works fine. Unfortunately though it seems OpenVPN client (at least RouterOS and NetworkManager on Linux implementations) don't forward hostname in SNI so I had to default to VPN and whitelist all https hostnames to route them to http proxy explicitly. And wildcard in map doesn't seem to work either :c
Carel over 6 years

I had to set proxy_protocol on; to get this working, perhaps others need to do the same...
Lochnair over 6 years

@Carel proxy_protocol is only needed here if you're using it on the upstream servers
vog over 5 years

This example selects by IP address. This is not what the question was all about, where a selection by SNI name is required. Of course everything is much easier if you can afford a separate IPv4 address per HTTPS domain.
Richard Kiefer almost 5 years

This answer is now out of date. See the other answers.
yegle almost 5 years

Would it be possible to specify the source IP for each of the backend?
rMili over 4 years

@Lochnair do we need to have SSL on vpn1.app.com, if so can it be a wild card one ?
Lochnair over 4 years

@rMili You do, and yes it can. At this stage the certificates haven't even been sent yet, so it doesn't matter.
Symon over 4 years

Even this answer looks good, it seems that OpenVpn client isn't sending the SNI, i've tried this approach, but i'm unable to make it work: stackoverflow.com/q/57713606/4175637
Khakhar Shyam over 4 years

@Lochnair I have http proxy in one upstream. It's working fine with your above config but in the proxy logs instead of logging the client ip address it logs the server ip. Can you help me how can i log client ip address?
Khakhar Shyam over 4 years

@RichardKiefer I have http proxy in one upstream. It's working fine with lochnair answer config but in the proxy logs instead of logging the client ip address it logs the server ip. Can you help me how can i log client ip address?
Richard Kiefer over 4 years

@KhakharShyam sorry, I am not fluent in that field. Consider posting a question regarding your specific problem.
turbophi over 4 years

It is very important to mention that the servers the upstreams are targeting must not use http2. Because if they do and share the same certificate (for example same wildcard certificate as my case was) then traffic that is supposed to reach another host will be sent through the same http2 connection/pipe. Causes
Paulo Boaventura over 3 years

the best answer i found you all my study a lot of gratitude @cnst
xtrinch over 3 years

I had to also do load_module /usr/lib/nginx/modules/ngx_stream_module.so; on top of the nginx conf file
Alex about 3 years

There is a good introduction to stream/tcp from the nginx folks in a blog post.
Vinay Mundada over 2 years

How does one pass the hostname to the upstream for non-tls data?
xeruf about 2 years

add a hostnames; line into the map to allow wildcard mappings ;)