Nginx - connect() failed (111: Connection refused) while connecting to upstream

Question

I am running a site that uses Django, Nginx, Gunicorn, Supervisord and fail2ban (which only allows ssh, http and https). The site is live and working correctly but there are some nginx error log entries that are concerning:

connect() failed (111: Connection refused) while connecting to upstream, client: x.x.x.x, server: www.example.com, request: "GET /example/url/to/get/ HTTP/1.1", upstream: "http://[::1]:8000/example/url/to/get/", host: "www.example.com"
upstream server temporarily disabled while connecting to upstream, client: x.x.x.x, server: www.example.com, request: "GET /example/url/to/get/ HTTP/1.1", upstream: "http://[::1]:8000/example/url/to/get/", host: "www.example.com"

Here is my nginx config:

upstream app_server_wsgiapp {
  server localhost:8000 fail_timeout=0;
}
server {
 listen 80;
 server_name www.example.com;
 return 301 https://www.example.com$request_uri;
}
server {
  server_name           www.example.com;
  listen                443 ssl;
  if ($host = 'example.com') {
    return 301 https://www.example.com$request_uri;
  }
  ssl_certificate       /etc/nginx/example/example.crt;
  ssl_certificate_key   /etc/nginx/example/example.key;
  ssl_session_timeout   1d;
  ssl_session_cache     shared:SSL:50m;
  ssl_protocols         TLSv1.1 TLSv1.2;
  ssl_ciphers           'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-
AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-
SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-
SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-
SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-
SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-
AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-
SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-
SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-
SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-
SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-
CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
  ssl_prefer_server_ciphers   on;
  access_log          /var/log/nginx/www.example.com.access.log;
  error_log           /var/log/nginx/www.example.com.error.log info;
  keepalive_timeout   5;
  proxy_read_timeout    120s;
  # nginx serve up static and media files
  location /static {
    autoindex on;
    alias /static/path;
  }
  location /media {
        autoindex on;
        alias /media/path;
  }
  location / {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    if (!-f $request_filename) {
        proxy_pass http://app_server_wsgiapp;
        break;
    }
  }
}

I do not have any errors in the Gunicorn logs.

Like I said, the site is working correctly. But I don't want to ignore error logs which could potentially become a bigger issue later.

Solution

Could this be because your system is dual-stack, but your upstream is IPv4 only?

It looks as if localhost is resolving to [::1], which depending on your upstream might be the problem in and of itself.

Given you are communicating over loopback, I would tend to assume the Connection refused is 'real' - it is reflective of the actual issue.

You can check whether this is the problem by replacing localhost with 127.0.0.1 in your upstream config:

upstream app_server_wsgiapp {
  server 127.0.0.1:8000 fail_timeout=0;
}

Comments

[darkpool]

Excellent, that appears to have fixed the issue. I unfortunately don't know enough about nginx to know why this fixed the issue, or why this was happening even with your explanation but im glad the issue is fixed. I will keep an eye on the logs for the rest of the day and then accept your answer once im sure it has been fixed.

[Hoylen]

The upstream service is only listening on the IPv4 address of 127.0.0.1 and is not listening on the IPv6 address of ::1.

[Hoylen]

When Nginx tries to resolve "localhost" into an IP address, it first gets ::1, tries connecting to the upstream service on ::1 and fails, writing the error message. Since that failed, it tries the next IP address it can find for "localhost", which happens to be 127.0.0.1 which works so that result is returned to the client. If you look inside your /etc/hosts file, you'll see there are more than one IP address for the name of "localhost".