Apache 2.4.7 mod_proxy_wstunnel tunneling too much (HTTP as well as WS)

Question

I'm running Apache 2.4.7 as a reverse proxy on Ubuntu 14.04 LTS. This Apache server acts as the entrypoint to a lot of different backend applications, which are accessed via different mod_proxy configurations in <Location> blocks

I need to provide reverse proxy access to an application that uses WebSockets. The application is a Java Spring application that serves HTML and other static files over HTTP, and then uses a WebSocket for dynamic data after the page has loaded.

I've got the application running behind Nginx using the following config:

location /newapp/ {
    proxy_pass http://newapp.example.com:8080/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

Unfortunately, due to needing an Apache authentication module that isn't available on Nginx, I can't use this in production.

What I want to do, in pseudo-Apache-config is:

<Location /newapp/>
    if not WebSockets:
        ProxyPass http://newapp.example.com:8080/
        ProxyPassReverse /
    else
        ProxyPass ws://newapp.example.com:8080/
        ProxyPassReverse /
</Location>

The Apache mod_proxy_wstunnel module makes me think that this should be possible. The WebSocket is accessed on the URL /api/socket/..., so I've tried separating the two types of ProxyPass using separate <Location> blocks:

<Location /newapp/>
    ProxyPass http://newapp.example.com:8080/ disablereuse=on
    ProxyPassReverse /

    ProxyPassReverseCookieDomain newapp.example.com apps.example.com
    ProxyPassReverseCookiePath http://newapp.example.com:8080/ /newapp/
</Location>

<Location /newapp/api/socket/>
    ProxyPass ws://newapp.example.com:8080/api/socket/
    ProxyPassReverse /
</Location>

This works initially - the browser requests http://apps.example.com/newapp/, the page loads over HTTP, the static assets are loaded, the JavaScript code connects to the websocket, everything is awesome.

However, when a new request is made over HTTP - say, for GET /newapp/static/someimage.png, something goes wrong. This request doesn't match against the WebSocket Location, so I would expect it to proxy GET /static/someimage.png through to http://newapp.example.com:8080/.

Instead, the application server receives a request for GET /newapp/static/someimage.png and returns a 404, as this isn't a URL that it's aware of. This breaks the application, as HTTP requests that should work fail instead.

Notes:

• This doesn't just occur for images - GET /newapp/api/ajax/someapicall also gets proxied through incorrectly.
• This doesn't always happen. While testing things to complete this question I managed to get the app to completely work. It might have been time based - I left the app running without interacting with it for a few minutes before making any new HTTP requests. When I did make new HTTP requests, they went through correctly.
• Disabling the <Location /newapp/api/socket/> section causes two things to happen - the WebSocket fails to connect, and the HTTP requests continue to work.
• I discovered this issue after refreshing the page via the browser's Refresh button. Instead of the page loading again, I saw the app's 404 screen.

What I think is happening:

I think mod_proxy_wstunnel, once activated by the first request to match /newapp/api/socket/, is taking over for all further inbound requests from the client, whether they match the Location or not. I tested this by adding a RequestHeader set Test "some_identifying_value" directive to each Location - the HTTP requests for static files & for /api/socket/info had the Test header on them, but the incorrectly-proxied HTTP requests did not have a Test header on them, which suggests that they are being passed straight through without being processed by the Apache directives.

Ultimately, my question is this: Is it possible to configure any version of Apache (I'm happy to upgrade!) to reverse-proxy WebSocket-based applications in such a way that HTTP requests are also reverse-proxied correctly after the WebSocket has connected? If so, how is this configured?

Solution 1

anders' answer got me 95% of the way there.

The basic scenario:

• We have a server on newapp.example.com
• Port 8080 is running both HTTP and WebSockets
• The URL that responds to WebSockets requests is /api/socket/
• We're reverse-proxying this application as http://apps.example.com/newapp/

This is how to configure WebSockets and HTTP reverse-proxying for the above scenario in a <Location> block:

<Location /newapp/>
    ProxyPass http://newapp.example.com:8080/
    ProxyPassReverse /

    RewriteEngine on
    RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
    RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
    RewriteRule /api/(.*) ws://newapp.example.com:8080/api/$1 [P]
</Location>

The final rewrite rule is crucial - without it, we'll pass the request /newapp/api/socket through to the WebSocket server - which it will reject.

The regex is parsing out everything after api - there might be a better way to capture that block, but this worked. We then have to remember to re-add /api/ to the final redirect URL.

Most importantly, HTTP requests continue to work after the WebSocket connection is established!

Solution 2

I'm using apache 2.4 as a proxy in front of my spring boot app, this app serves some rest api calls, websockets (sockjs) and some static pages. I had some problems getting the websocket to work, the secret there was to add the rewrite rules you see below, now it works for me, my apache virtual host config looks like this:

<VirtualHost *:443>
  SSLEngine on
  SSLCertificateFile /etc/httpd/ssl/my.crt
  SSLCertificateKeyFile /etc/httpd/ssl/my.key
  SSLCertificateChainFile /etc/httpd/ssl/intermediate.crt
  ProxyPreserveHost On
  ProxyPass / http://127.0.0.1:8080/
  ProxyPassReverse / http://127.0.0.1:8080/
  ProxyRequests Off
  RewriteEngine on
  RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
  RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
  RewriteRule .* ws://localhost:6868%{REQUEST_URI} [P]
  ServerName my.dnsname.com
</VirtualHost>

Comments

[mettjus]

Firstly: thank you very much for reading this! I really appreciate it :)

[mettjus]

Secondly: AAARGH this is driving me nuts!

[mettjus]

This was so close! Thank you so much! I had to make a few changes to make it work in a <Location> block, though