Django gunicorn nginx (111: Connection refused) while connecting to upstream

Question

A Django application is running on the AWS instance, configured via gunicorn and nginx, it is running well for more than a year, but suddenly, I got 502 bad gateway error, then I saw the below mentioned message in the nginx error log,

2017/05/17 16:18:35 [error] 1040#0: *7460 connect() to unix:/home/ubuntu/webapps/myproject/myproject/myproject.sock failed (111: Connection refused) while connecting to upstream, client: xx.xxxx.xx.xxx, server: xx.xx.xx.xxx, request: "GET / HTTP/1.1", upstream: "http://unix:/home/ubuntu/webapps/myproject/myproject/myproject.sock:/", host: "xx.xx.xx.xxx", referrer: "http://xx.xx.xx.xxx"

my nginx configuration:

server {
        client_max_body_size 200M;
        listen 80;
        listen [::]:80 ipv6only=on;
        server_name xx.xx.xx.xxx;
        listen 443 ssl;
        ssl_certificate /etc/nginx/ssl/myserver.crt;
        ssl_certificate_key /etc/nginx/ssl/myserver.key;


        location = /favicon.ico { access_log off; log_not_found off; }
        location /static/ {
                root /home/ubuntu/webapps/myproject/myproject;
        }

        location / {
                include proxy_params;
                proxy_pass http://unix:/home/ubuntu/webapps/myproject/myproject/myproject.sock;
                proxy_set_header X-Forwarded-Protocol $scheme;
        }

        if ($scheme = http){
                return 301 https://xx.xx.xx.xxx$request_uri;
        }

        if ($http_host = pas-cash.com){
                return 303 https://xx.xx.xx.xxx$request_uri;
        }
}

my gunicorn.conf

description "Gunicorn application server handling myproject"

start on runlevel [6789]
stop on runlevel [!6789]

respawn
setuid ubuntu
setgid www-data
chdir /home/ubuntu/webapps/myproject/myproject

exec /home/ubuntu/webapps/myproject/venv/bin/gunicorn --workers 3 --bind unix:/home/ubuntu/webapps/myproject/myproject/myproject.sock myproject.wsgi:application

After that I restarted the nginx by following command

sudo service nginx restart

After restarting, the application is running well, I can't find what will be the specific reason behind this error, I googled it for this, but I got different types of answer but nothing suitable for me, can you guys please help me out for, why this happened, is there any thing missing in my configuration or what will the common/general reason behind this behavior. It will be very helpful for me, Thanks in advance.

Solution 1

try to remove http:// from the proxy_pass in the nginx configuration:

server {
    client_max_body_size 200M;
    listen 80;
    listen [::]:80 ipv6only=on;
    server_name xx.xx.xx.xxx;
    listen 443 ssl;
    ssl_certificate /etc/nginx/ssl/myserver.crt;
    ssl_certificate_key /etc/nginx/ssl/myserver.key;


    location = /favicon.ico { access_log off; log_not_found off; }
    location /static/ {
            root /home/ubuntu/webapps/myproject/myproject;
    }

    location / {
            include proxy_params;
            proxy_pass unix:/home/ubuntu/webapps/myproject/myproject/myproject.sock;
            proxy_set_header X-Forwarded-Protocol $scheme;
    }

    if ($scheme = http){
            return 301 https://xx.xx.xx.xxx$request_uri;
    }

    if ($http_host = pas-cash.com){
            return 303 https://xx.xx.xx.xxx$request_uri;
    }
}

The reason is that gunicorn is listening on a unix socket (the --bind argument). Then nginx should forward traffic to this socket. http:// stands for a TCP socket in a regular IP:PORT, which is not your case.

Solution 2

This is caused "suddenly" not because of an nginx error but rather an error with gunicorn or your app (code error, packages not installed etc.). It should be relatively easy to log and fix though.

First try running your app from the server python manage.py runserver and see if you get any issues. The same for ... migrate. Often the issue that production does not work but local does is because of missing packages or missing migrations. Create a requirements.txt file on local and install it on production.

If the error is still there check the gunicorn logs with gunicorn --log-file=- YourApp.wsgi:application . Once all those errors have been corrected run

sudo systemctl status gunicorn.socket
sudo systemctl status gunicorn

And you want to have both active and running. If you start getting a 400 error that is a good sign as it is now a Django error (usually allowed hosts). Turn debug=True to see the exact error from django.

Remember whenever any changes are made to code run

sudo systemctl daemon-reload
sudo systemctl restart gunicorn

Just FYI if none of the above work then you can always check your nginx logs with

sudo tail -30 /var/log/nginx/error.log

Comments

[Ajay Kumar]

Thanks for your explanation. I can understand the problem. Can you please tell me what are the possible reason that nginx is not forwarding the traffic to the socket(like, any network interruption or some thing like this.)

[alfonso.kim]

Hum, then is still not working? Please make sure that gunicorn and nginx are both up and running

[Hassan Baig]

Useful to be reminded of the fundamentals.