11: Resource temporarily unavailable, while connecting to upstream + Bad Gateway (Nginx)

Question

For all the sites on my new server, it seems as though once nginx has served some amount of traffic, it starts serving "ERROR 502 - Bad Gateway!".

Let's take a look at some logs and config...

• OS: Ubuntu 16.04 (Server)

• Hardware: 6 cores, 12GB RAM

• Panel: ISPConfig

• nginx version: 1.10.3

From /var/www/example/log/error.log:

11: Resource temporarily unavailable) while connecting to upstream, client: 216.170.123.179, server: example.com, request: "GET /KHFxygk9 HTTP/1.1", upstream: "fastcgi://unix:/var/lib/php7.0-fpm/web3.sock:", host: "example.com", referrer: "http://example.com/"
2017/08/10 06:01:06 [error] 15408#15408: *31372 connect() to unix:/var/lib/php7.0-fpm/web3.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 192.3.114.12, server: example.com, request: "GET /jiPNQjwp HTTP/1.1", upstream: "fastcgi://unix:/var/lib/php7.0-fpm/web3.sock:", host: "example.com", referrer: "http://example.com/"

Here are my config files:

• /etc/php/7.0/fpm/pool.d/www.conf

---

[www]    
user = www-data
group = www-data

listen = /run/php/php7.0-fpm.sock

listen.owner = www-data
listen.group = www-data

pm = dynamic
pm.max_children = 12    
pm.start_servers = 2    
pm.min_spare_servers = 1    
pm.max_spare_servers = 3    
pm.max_requests = 500

• /etc/nginx/nginx.conf

---

user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
        worker_connections 768;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";


        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

systemctl status php7.0-fpm

● php7.0-fpm.service - The PHP 7.0 FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php7.0-fpm.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2017-08-10 04:44:16 UTC; 1h 30min ago
  Process: 20310 ExecReload=/bin/kill -USR2 $MAINPID (code=exited, status=0/SUCCESS)
  Process: 20302 ExecReload=/usr/lib/php/php7.0-fpm-checkconf (code=exited, status=0/SUCCESS)
  Process: 13060 ExecStartPre=/usr/lib/php/php7.0-fpm-checkconf (code=exited, status=0/SUCCESS)
 Main PID: 13070 (php-fpm7.0)
   Status: "Processes active: 20, idle: 12, Requests: 6874, slow: 0, Traffic: 0.3req/sec"
   CGroup: /system.slice/php7.0-fpm.service
       ├─13070 php-fpm: master process (/etc/php/7.0/fpm/php-fpm.conf)                      
       ├─13072 php-fpm: pool apps-{fpm_domain}                                              
       ├─13073 php-fpm: pool apps-{fpm_domain}                                              
       ├─13074 php-fpm: pool ispconfig                                                      
       ├─13075 php-fpm: pool ispconfig                                                      
       ├─13076 php-fpm: pool web1                                                           
       ├─13077 php-fpm: pool web1                                                           
       ├─13079 php-fpm: pool web2                                                           
       ├─13080 php-fpm: pool web3                                                           
       ├─13081 php-fpm: pool web3                                                           
       ├─13082 php-fpm: pool web4                                                           
       ├─13083 php-fpm: pool web4                                                           
       ├─13084 php-fpm: pool web5                                                           
       ├─13085 php-fpm: pool web5                                                           
       ├─13087 php-fpm: pool www                                                            
       ├─13088 php-fpm: pool www                                                            
       ├─16186 php-fpm: pool web3                                                           
       ├─16603 php-fpm: pool web3                                                           
       ├─16604 php-fpm: pool web3                                                           
       ├─16605 php-fpm: pool web3                                                           
       ├─16609 php-fpm: pool web3                                                           
       ├─16614 php-fpm: pool web3                                                           
       ├─16629 php-fpm: pool web3                                                           
       ├─16630 php-fpm: pool web3                                                           
       ├─18019 php-fpm: pool web2                                                           
       ├─18026 php-fpm: pool web2                                                           
       ├─18027 php-fpm: pool web2                                                           
       ├─18028 php-fpm: pool web2                                                           
       ├─18369 php-fpm: pool web2                                                           
       ├─18370 php-fpm: pool web2                                                           
       ├─18410 php-fpm: pool web2                                                           
       ├─18411 php-fpm: pool web2
       └─18413 php-fpm: pool web2                                                           
Aug 10 04:44:16 web1.example.com systemd[1]: Stopped The PHP 7.0 FastCGI Process Manager.
Aug 10 04:44:16 web1.example.com systemd[1]: Starting The PHP 7.0 FastCGI Process Manager...
Aug 10 04:44:16 web1.example.com systemd[1]: Started The PHP 7.0 FastCGI Process Manager.

Solution 1

These errors could be caused by lack of allowed maximum number of socket connections (mostly default is 128). You can view the current limit by executing cat /proc/sys/net/core/somaxconn and it's highly recommended to increase this limit:

sudo echo "net.core.somaxconn = 65535" | tee -a /etc/sysctl.conf
sudo sysctl -p

And check whether it has changed cat /proc/sys/net/core/somaxconn.

Also these errors could be caused by lack of number of incoming connections backlog queue (mostly default is 1000). Check the current limit executing cat /proc/sys/net/core/netdev_max_backlog. It's recommended to increase the limit:

sudo echo "net.core.netdev_max_backlog = 65535" | tee -a /etc/sysctl.conf
sudo sysctl -p

And check whether it has changed cat /proc/sys/net/core/netdev_max_backlog

Note: it's impossible to follow my advice if these troubles appeared on your OpenVZ VPS, because you can't change kernel parameters using this type of virtualization. The better option in this case is trying to use port listening in your php-fpm pool config instead of socket (e.g. listen = 9000). Check your busy ports for finding available one by executing sudo netstat -tunlp.

Solution 2

you have some mismatch in cfg, in php define listen = /run/php/php7.0-fpm.sock but nginx try to connect to fastcgi://unix:/var/lib/php7.0-fpm/web3.sock. Set the same value on both places (php and nginx)

Comments

[Vijay Anand Pandian]

Have you fixed this issue?

[Zenexer]

If max_children has to be increased much higher than that, it's either a very poorly designed application, or the server has a lot of cores.

[SomeDudeSomewhere]

This was good, it brought me the following results: status: 1xx=0 2xx=10000 3xx=0 4xx=0 5xx=0 From... status: 1xx=0 2xx=8136 3xx=0 4xx=0 5xx=1864 However, seeing some errors now like these on nginx errors, what do you think Alex? failed (24: Too many open files) while connecting to upstream,

[Bappi Saha]

How can you test the current usage towards these limits, in order to know which limit is being hit?

[Freedo]

Is it possible to raive the values more than 65535 ?