Nginx 502 Bad Gateway - 111: Connection refused, port not being listened to

Question

I'm aware there are many other similar questions but the solutions from those questions I feel I've already got in place.

I've got 3 sites running locally on a VirtualBox Ubuntu 12.04.4 LTS Server, I'm using nginx and php-fpm.

They are configured to run on ports 9001 -> 9003:

server:/etc/php5/fpm/pool.d$ grep 900 *
my-app-deploy.conf:listen = 127.0.0.1:9002
my-app-dev.conf:listen = 127.0.0.1:9001
my-web.conf:listen = 127.0.0.1:9003

server:/etc/nginx/conf.d$ grep 900 *
my-app-deploy.conf:                fastcgi_pass   127.0.0.1:9002;
my-app-dev.conf:                fastcgi_pass   127.0.0.1:9001;
my-app-web.conf:                fastcgi_pass   127.0.0.1:9003;

I've restarted both php-fpm and nginx successfully:

server:$ sudo service nginx restart
 * Restarting nginx nginx
   ...done.
server:$ sudo service php5-fpm restart
 * Restarting PHP5 FastCGI Process Manager php5-fpm
   ...done.

But if I look at what ports are being listened to, I only see 9002 and 9003:

server:$ sudo lsof -i | grep 900
php5-fpm  1020     root    7u  IPv4   8557      0t0  TCP localhost:9002 (LISTEN)
php5-fpm  1020     root    8u  IPv4   8558      0t0  TCP localhost:9003 (LISTEN)
php5-fpm  1021 www-data    0u  IPv4   8557      0t0  TCP localhost:9002 (LISTEN)
php5-fpm  1022 www-data    0u  IPv4   8557      0t0  TCP localhost:9002 (LISTEN)
php5-fpm  1023 www-data    0u  IPv4   8557      0t0  TCP localhost:9002 (LISTEN)
php5-fpm  1024 www-data    0u  IPv4   8557      0t0  TCP localhost:9002 (LISTEN)
php5-fpm  1025 www-data    0u  IPv4   8558      0t0  TCP localhost:9003 (LISTEN)
php5-fpm  1026 www-data    0u  IPv4   8558      0t0  TCP localhost:9003 (LISTEN)
php5-fpm  1027 www-data    0u  IPv4   8558      0t0  TCP localhost:9003 (LISTEN)
php5-fpm  1028 www-data    0u  IPv4   8558      0t0  TCP localhost:9003 (LISTEN)

And I get the following error from nginx:

server:~/vhosts/my-app/logs$ tail -n 1 nginx/error.log 
2014/06/20 11:07:36 [error] 2434#0: *4 connect() failed 
    (111: Connection refused) while connecting to upstream, client: 192.168.0.10,
    server: my.app.dev, request: "GET /api/test.php HTTP/1.1", upstream:
    "fastcgi://127.0.0.1:9001", host: "my.app.dev"

I'm hoping I've just missed something really basic, but when I added the site on 9002 I don't remember having to do anything else apart from add the new port number to both the conf files

UPDATE The site root is actually serving, it's only the api/test.php page that is returning 502

UPDATE I moved the ports around, I swapped web and dev around so dev is now on the 9001 port, but that's made no difference. I've updated the question to reflect this.

All 3 ports are serving content - I think it's something to do with redirects I've got setup for the API section.

Here is a diff between the dev and deploy PHP and nginx configs:

server:/etc/nginx/conf.d$ diff my-app-dev.conf my-app-deploy.conf 
3c3
<         server_name my.app.dev;
---
>         server_name my.app.deploy;
10c10
<         root /var/www/vhosts/my-app/public_html/mobile;
---
>         root /var/www/vhosts/my-app/public_html/mobile_deploy;
47c47
<                 fastcgi_pass   127.0.0.1:9001;
---
>                 fastcgi_pass   127.0.0.1:9002;

server:/etc/php5/fpm/pool.d$ diff my-app-dev.conf my-app-deploy.conf
33c33
< listen = 127.0.0.1:9001
---
> listen = 127.0.0.1:9002
384c384
< php_admin_value[doc_root] = /var/www/vhosts/my-app/public_html/mobile
---
> php_admin_value[doc_root] = /var/www/vhosts/my-app/public_html/mobile_deploy

And here is the complete conf file for the dev site, on port 9001, that is giving me the 502

server {
        listen 80;
        server_name my.app.dev;

        error_log  /var/www/vhosts/logs/my-app/nginx/error.log;
        access_log /var/www/vhosts/logs/my-app/nginx/access.log  main;

        error_page 404 /404;

        root /var/www/vhosts/my-app/public_html/mobile;
        index index.html index.php;

        client_max_body_size 10M;

        proxy_read_timeout 180s;

        # deny any attempt to access hidden files
        location ~ /\. { deny all; }

        # disable logging for favicon
        location = /favicon.ico {
                log_not_found off;
        }

        location ~* \/api\/[0-9]\/.*\.zip {
                try_files $uri /api/3/nomarket.data.zip;
        }

        location ~* \/api\/picture\/[a-z]+\/? {
                if (!-e $request_filename) {
                        rewrite ^/api/picture/([a-z]+)/? /api/picture.php?type=$1 last; break;
                }
        }

        location /api {
                if (!-e $request_filename) {
                        rewrite ^/api/(.*)$ /api/router.php?rest_data=$1 last; break;
                }
        }

         # allow php files to be executed
        location ~ \.php$ {
                # only call php if the file exists. Very important for security!
                try_files  $uri =404;
                include /etc/nginx/fastcgi_params;
                fastcgi_pass   127.0.0.1:9001;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_index  index.php;
        }

        # just serve static files, please.
        location / { }
}

Solution 1

Your lsof output looks fairly conclusive that php-fpm is not listening on port 9001 (though you might like to use sudo netstat -plnt for a more concise listing that's faster to generate).

Presumably then, the issue is with php-fpm. I'd suggest you forget about nginx for the moment, and use telnet localhost 9001 for debugging. I expect you'll see a failure to connect. Go back to using nginx once you can get a connection.

I'm guessing that either you have a configuration issue with the site that's supposed to be on port 9001, or php-fpm was for some reason unable to bind to that port.

Either way, the first thing I'd look for is log entries from php-fpm at startup time. You might need to enable those logs. See this question for that: How to configure php-fpm to log logs to log files configured in nginx virtualhosts

Those config file diffs look straight-forward enough. Maybe check for control characters on those lines (eg Windows style line ends can creep in with some work practices). Also, remember to check for differences in file permissions as well as file content.

If the problem relates to binding to the port, then using a different port is likely to change the behaviour. Worth a shot, even if just to rule that out. You could also look through strace output to see what happens there if/when fpm attempts to bind port 9001. Eg strace -p [pid] | grep -n 10 9001 .

Solution 2

PHP FPM Pool Names

From your diff it appears you have duplicated PHP pool names in /etc/php-fpm.d. I am pretty certain only the configuration file that loads last will be in effect.

As a result, with duplicate pool names only the last loaded name/port pair will spin up.

Looking at your data, perhaps my-app-dev.conf and my-app-deploy.conf share the same pool name since it is not shown in the diff. As a result, only one of these pools is in effect.

I recommend double checking your php pool names.

Comments

[Michael Hampton]

Your pool on port 9003 isn't running. Post its configuration.

[Pete]

Do you mean the full php-fpm conf from /etc/php5/fpm/pool.d?

[garethTheRed]

502 means there's a problem 'upstream'. Can you access the site on 127.0.0.1:9001 (or whichever port it's on now)?

[Pete]

What do you mean by access it? Ping it from the Virtual Box server? I can access the site from Chrome on the host system

[ADM]

there will be an error like :ERROR: An another FPM instance seems to already listen on ERROR: FPM initialization failed

[Pete]

Sorry I thought I'd commented on this, it was just an error typing up my question, that wasn't the problem.

[Pete]

So what actually did it was I'd left the name of the server in both php-fpm conf files as my.app.dev - changing the deploy conf server name to my.app.deploy got them both working. If you look in the diffs I posted, nginx shows different server names but nothing comes up in the diff for php-fpm about the server name. So thanks @mc0e for narrowing it down to a problem with php, and I moved it to port 9004 which didn't help, confirming it wasn't that port in particular - I just went through the php configs line by line to spot it

[Robust]

for anyone who had my problem i'll write it here: I had the bad gateway problem, and it ended up being because I had changed pm = static to pm = ondemand and the ondemand setting didn't even exist yet in my installed version of nginx.