How to run Gunicorn upstream with an Nginx SSL configuration?

Question

I have an nginx server with an SSL certificate installed. I want to pass any requests upstream to my Gunicorn server running at 0.0.0.0:8000. However, whenever I run the Gunicorn server, it gives me an error saying that there's too many redirect loops. If I run gunicorn over https, then the connection WILL become secure, but it won't connect to the gunicorn server and it'll just say bad gateway. Also, here is the error I get when attempting to connect while running gunicorn with https:

Traceback (most recent call last):
  File "/opt/bitnami/python/lib/python2.7/site-packages/gunicorn/arbiter.py", line 515, in spawn_worker
    worker.init_process()
  File "/opt/bitnami/python/lib/python2.7/site-packages/gunicorn/workers/base.py", line 126, in init_process
    self.run()
  File "/opt/bitnami/python/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 119, in run
    self.run_for_one(timeout)
  File "/opt/bitnami/python/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 66, in run_for_one
    self.accept(listener)
  File "/opt/bitnami/python/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 30, in accept
    self.handle(listener, client, addr)
  File "/opt/bitnami/python/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 141, in handle
    self.handle_error(req, client, addr, e)
  File "/opt/bitnami/python/lib/python2.7/site-packages/gunicorn/workers/base.py", line 213, in handle_error
    self.log.exception("Error handling request %s", req.uri)
AttributeError: 'NoneType' object has no attribute 'uri'
[2016-01-01 15:37:45 +0000] [935] [INFO] Worker exiting (pid: 935)
[2016-01-01 20:37:45 +0000] [938] [INFO] Booting worker with pid: 938
[2016-01-01 15:37:46 +0000] [938] [ERROR] Exception in worker process:

Here is my nginx configuration:

  server {
    # port to listen on. Can also be set to an IP:PORT
    listen 80;
    listen 443 ssl;
    ssl_certificate /etc/ssl/pyhub_crt.crt;
    ssl_certificate_key /etc/ssl/pyhub.key;
    server_name www.pyhub.co;
    server_name_in_redirect off;
    access_log /opt/bitnami/nginx/logs/access.log;
    error_log /opt/bitnami/nginx/logs/error.log;
    location /E0130777F7D5B855A4C5DEB138808515.txt {
        root /home/bitnami;
    }

    location / {
    proxy_pass_header Server;
    proxy_set_header Host $host;
    proxy_set_header X-Scheme $scheme;
    proxy_set_header X-SSL-Protocal $ssl_protocol;
    proxy_connect_timeout 10;
    proxy_read_timeout 10;
    proxy_redirect http:// $scheme://;
    proxy_pass http://localhost:8000;
    }

Solution 1

If gunicorn is bound to 0.0.0.0 it is bound to all interfaces, therefore it is already exposed to the outside interface. If nginx tries to bind to the any interface on the same port it will fail.

Gunicorn should be bound to a specific ip or better 127.0.0.1 so it is only bound to the internal ip.

Sesond, you say you want to pass https to gunicorn, but the traffic is protected with SSL to your proxy which has the certificates, that is ngninx. After that, the traffic is in clear internally (i.e. it is http) to gunicorn, unless you also have SSL setup on gunicorn.

So, your nginx config should have:

• An upstream server to gunicorn with ip 127.0.0.1 port 8080 or whatever you want.
• A server directive listening on port 80 for http and 443 for https
• a proxy-pass directive in the server bloc to forward to upstream

my nginx proxy config for SSL is something like this:

upstream website    {
    ip_hash;                        # for sticky sessions, more below
    server                          website:8000 max_fails=1 fail_timeout=10s;
}

server {
    # only listen to https here
    listen                          443 ssl http2;
    listen                          [::]:443 ssl http2;
    server_name                     yourdomain.here.com;

    access_log                      /var/log/nginx/yourdomain.here.com.access.log;
    error_log                       /var/log/nginx/yourdomain.here.com.error.log;
    ssl                             on;
    ssl_certificate                 /etc/nginx/certs/ca-cert.chained.crt;
    ssl_certificate_key             /etc/nginx/certs/cert.key;
    ssl_session_cache               shared:SSL:5m;
    ssl_session_timeout             10m;
    ssl_protocols                   TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers       on;
    #ssl_dhparam                     /etc/nginx/certs/dhparams.pem;
    # use the line above if you generated a dhparams file 
    ssl_ciphers                     'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
    ssl_buffer_size                 8k;

    location / {
        proxy_pass                  http://website;
        proxy_set_header            Host $host;
        proxy_set_header            X-Real-IP $remote_addr;
        proxy_http_version          1.1;

        proxy_set_header            X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header            X-Forwarded-Proto http;
        proxy_redirect              http:// $scheme://;
    }
}

# redirect http to https here
server {
    listen                          80;
    listen                          [::]:80;
    server_name                     yourdomain.here.com;
    return                          301 https://$server_name/;
}

Solution 2

Try this:

upstream app_server {
    server 127.0.0.1:8000 fail_timeout=0;
}

server {
    # port to listen on. Can also be set to an IP:PORT
    listen 80;
    listen 443 ssl;

    ssl_certificate /etc/ssl/pyhub_crt.crt;
    ssl_certificate_key /etc/ssl/pyhub.key;

    server_name www.pyhub.co;
    server_name_in_redirect off;

    access_log /opt/bitnami/nginx/logs/access.log;
    error_log /opt/bitnami/nginx/logs/error.log;

    location /E0130777F7D5B855A4C5DEB138808515.txt {
        root /home/bitnami;
    }

    location / {
        try_files @proxy_to_app;
    }

    location @proxy_to_app {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;

        proxy_pass http://app_server;

        proxy_pass_header Server;
        proxy_set_header Host $host;
        proxy_set_header X-Scheme $scheme;
        proxy_set_header X-SSL-Protocol $ssl_protocol;
    }
}

Note that this will (properly) terminate SSL at nginx rather than gunicorn. You also misspelled Protocal in X-SSL-Protocal; fixed in my example :)

See here for canonical information on deploying gunicorn with nginx.

Solution 3

I should have posted this answer a long time ago, as I've had my site working for quite a while now. But here's how I got my configuration to work.

After fiddling around a lot with the Nginx configuration, I got pretty frustrated and gave up. So, I deleted that server, made a new one, with fresh installations of everything, and cloned the source code for my website from the repository that hosts it. After doing this, I used this Nginx configuration for it to work:

  upstream djangosite {
    server 127.0.0.1:8000 fail_timeout=2s;
  }
  server {
    # port to listen on. Can also be set to an IP:PORT
    listen 443 ssl;
    server_name pyhub.co;
    ssl_certificate /etc/ssl/pyhub.crt;
    ssl_certificate_key /etc/ssl/pyhub.key;
    location / {
      proxy_pass http://djangosite;
    }
  server {
    listen 80;
    server_name pyhub.co;
    return 301 https://$server_name$request_uri;
  }

To this day I still don't know what was causing me problems with my configuration, or why I had to get another server with a completely fresh install on it, but I'm just glad it works. I would select one of the two excellent answers already provided, but none of them gave me the solution, and I don't want to select my own, because I believe it is too specific of a problem that I had and selecting this answer might not be able to help many people in the future.

Comments

[gxx]

I'm having a hard time to understand your problem: Could you please clearly state what works, what doesn't, and what you want to achieve?

[Dorian Dore]

The nginx ssl connection works. However, I want it to pass to the upstream gunicorn server, but it is not working. This is what I want to achieve.

[gxx]

You want to pass http or https to gunicorn?

[Dorian Dore]

I want to pass https to Gunicorn.

[gxx]

If you're hosting both ngnx and gunicorn on the same machine, then this doesn't make much sense, but well...could you show your gunicorn config / the command you're using to start gunicorn?

[Dorian Dore]

gunicorn -b 0.0.0.0:8000 myproject.wsgi

[gxx]

Two things to note regarding your gunicorn config: I guess you don't want it to bind to 0.0.0.0, which means all interfaces on your machine, thus it's reachable directly from the public internet, better bind to 127.0.0.1. Regarding https: You didn't enable this, so how do you think should this work? If you still want to enable https at gunicorn (even it's a bit pointless and creates just more work [in your environment]), have a look at this; if not, have a look at the answer of Will.

[Dorian Dore]

Thank you for the reply! I tried your suggestion, but it is still telling me that I have too many redirect loops.

[MrE]

added my config file for you to look at.

[Dorian Dore]

Alright, I was able to get my site to work! However, it'll only work if one connects via HTTPS, it doesn't work with HTTP. If you try to connect over HTTP it'll just give you a Bad Request error. I tried adding SECURE_SSL_REDIRECT to my settings.py file, but then it just gives me the too many redirect loops error.

[MrE]

did you use the # redirect http to https here server { listen 80; listen [::]:80; server_name yourdomain.here.com; return 301 https://$server_name/; } part that is at the bottom of the file I shared? you need to scroll down to see the whole thing

[Dorian Dore]

I didn't but even after adding it, I'm still getting the bad request.

[MrE]

Are you listening to 443 only on the main server directive? Or did you keep 80 as well? Remove 80, let 80 be redirected to 443 with the bottom part, otherwise you are listening on 80 but providing a cert to validate and that will fail. If you are still hav ok ng troubles can you post the result of a curl request in verbose mode curl -vvv yourdomain.com

[Dorian Dore]

I'm still getting the bad request, I've added a log file for gunicorn, and it's not even outputting an error, it's jsut returning a 400.

[MrE]

can you post the curl output?

[Dorian Dore]

* Rebuilt URL to: www.pyhub.co/ * Hostname was NOT found in DNS cache * Trying 52.7.19.183... * Connected to www.pyhub.co (52.7.19.183) port 80 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.38.0 > Host: www.pyhub.co > Accept: */* > < HTTP/1.1 400 OK < Date: Tue, 05 Jan 2016 02:03:06 GMT < Content-Type: text/html; charset=utf-8 < Transfer-Encoding: chunked < Connection: keep-alive * Server gunicorn/19.4.3 is not blacklisted < Server: gunicorn/19.4.3 < X-Frame-Options: SAMEORIGIN <

[MrE]

So, nginx is doing its job and sending the request to you gunicorn server. It's obviously gunicorn that is then not liking the request as it partially renders your pyhub page.

[MrE]

try this one: proxy_set_header X-Forwarded-Proto https; https instead of http in the X-Forward_Proto. this is what gunicorn read the docs seem to say docs.gunicorn.org/en/stable/deploy.html#nginx-configuration

[Dorian Dore]

I'm still getting the 400.

[MrE]

The error showing up on your post is in the python code. Can you try to trace it to figure why the expected uri parameters is not in the request? Is it in the request with http? nginx is not forwarding the headers properly somehow and you need to find which part is different

[Dorian Dore]

I'm not sure what could be causing the above error in Gunicorn, I think it might be a bug, and I've opened an issue on the git repo accordingly. I don't think nginx is causing that bug because I've tried running gunicorn with nginx stopped and i still get that error.