502 Bad Gateway error for my server running with Node JS on nginx proxy

Question

I am getting 502 bad gateway error: when I check the nginx error log I find this:

2017/05/06 02:36:04 [error] 48176#0: *135 connect() failed (111: Connection refused) while connecting to upstream, client: 10.163.XX.X, server: abc-def-ghi, request: "GET /favicon.ico HTTP/1.1", upstream: "https://127.0.0.1:5300/favicon.ico", host: "hostnname", referrer: "hostname-1

I searched internet enough but could not find anything. One thing to note here is that, this intermittent error is coming only on a particular page.

Could this be a code issue? or nginx configuration issue> Can anyone please help me here.

Some of my nginx conf:

  upstream node_api_server {
    server localhost:5300 fail_timeout=0;
  }

location / {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Host $http_host;
    proxy_set_header X-NginX-Proxy true;
    proxy_read_timeout 5m;
    proxy_connect_timeout 5m;
    proxy_pass_header Set-Cookie;

    proxy_pass https://node_api_server;
    proxy_redirect off;
    proxy_buffer_size   128k;
    proxy_buffers   4 256k;
    proxy_busy_buffers_size   256k;
    break;
}

Solution 1

502 errors are generally caused by NGINX being unable to pass a request to "upstream", in this case your Node.js server (which is also what the error message suggests: "Connection refused"").

It may be crashing and restarting, so check its logfiles to see what's causing the crashes.

Solution 2

I'm a little bit of a noob to this stuff, but I spent hours trying to debug my issue. It ended up being that my global variable for the port in my API wasn't being assigned correctly so node wasn't listening to the right port. Due to my infrequent use of a lot of this stuff, I wrote down how I should trouble-shoot it in the future and figured I would share. Again, this is for my API, so you may be slightly different (I was using port 3006).

• In any console, double check you are getting an error by running wget https://example.com/api/v1/ping or if you are ssh'd into the production console you can do wget 127.0.0.1:3006/api/v1/ping
• Make sure that node is running properly. After starting node in pm2 using npm run build then npm run start, make sure pm2 is running correctly with pm2 status
• Check the error logs for pm2 with pm2 logs. This should show the normal output from console.log messages in your API. Does it say it is running on the correct port? You can have your api log your port at startup to confirm it is on the correct port. Check other errors as well. You can clear all logs in pm2 using pm2 flush
• Now make sure that your server is listening to incoming traffic on the port that you specified sudo netstat -plunt will give you a list of ports that are open and the programs that are using them.
• You can get more info on the node apps using sudo ps aux | grep node to get all node programs listening to ports and it will show the full path for the node file path. You can match the PID in the previous command to make sure your ports are matching.
• Check the status of nginx to make sure it is running: sudo systemctl nginx status or replace status with start to get it started
• Now make sure that your nginx ports are using that incoming port. Check the nginx error log with sudo vim /var/log/nginx/error.log to confirm your port and server name are correct. Every time you send your server a command it will record the 502 error here with an error message: 2021/02/20 03:05:28 [error] 26646#26646: *644 connect() failed (111: Connection refused) while connecting to upstream, client: XXX.XXX.XXX.XX, server:example.com, request: "GET /api/v1/ping HTTP/1.1", upstream: "http://127.0.0.1:3006/api/v1/ping", host: "example.com"
• You can check the status of your nginx config files to make sure you don’t have any errors in the config by doing sudo nginx -t
• Check the Nginx config file in /etc/nginx/nginx.config. This file probably includes the sites-enabled/default config file. Mind as well check the sites-available/default config file as well to make sure all ports and server names are correct and making sure there are no duplicates (most likely would show up in the error logs).
• For any changes that are made to the config files, you need to restart the nginx system using sudo service nginx restart

Hopefully that helps some of you! :) Happy Coding

Solution 3

In my case, I received the same error, but when I check my logs of ingress-nginx. I found this

upstream sent too big header while reading response header from upstream,
 client: xxx.xxx.xxx.xx, server: sample-site.domain, request:

to overcome this issue I simply increased the size of --max-http-header-size of the node. and I was able to fix my issue.

Comments

[Saurav Bajracharya]

Hi @robertklep Can you tell me where to find logfiles?

[robertklep]

@SauravBajracharya that entirely depends on the way nginx is configured, but check /var/log/nginx/

[Andrew]

Just to add I had this problem. Two calls were made to the node server consecutively and the first one had an error causing the server to restart. The second call returned the 502 error because the node server was in the process of restarting. Spent a lot of time checking ports etc when this was it all along. Thanks.

[JustCurious]

Thanks for posting this answer. Gave me a sudden laugh. :)