Nginx error recv() failed (104: Connection reset by peer)

Question

Since a couple of days ago, I'm getting some errors on my server. I use CentOS 6.5 with Parallels 12.0.18, Apache server to serve dynamic content and Nginx as proxy to serve static content.

At first, I was getting the following error:

[error] 29951#0: *5138862 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 89.7.24.108, server: , request: "GET /page/2/ HTTP/1.1", upstream: "http://ip:7080/page/2/", host: "domain.es", referrer: "http://domain.es/"

Then, I changed some configuration, like increasing MaxClients on my "httpd.conf" file and this lines to my /etc/nginx/conf.d/timeout.conf file:

proxy_connect_timeout       600;
proxy_send_timeout          600;
proxy_read_timeout          600;
send_timeout                600;

It all seemed to be working fine until I got the same errors again, along with a new one:

[error] 15228#0: *130292 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 89.130.25.154, server: domain.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:7080/", host: "domain.com"

I have two different websites on the same server. That's why you see two different hosts in there.

Here's the problem: when I got these errors, I got a "502 Bad Gateway" and the server becomes so slow that I can't even log in using the SSH terminal. I can only fix it temporally by resetting the httpd service.

I know there are other topics similar to this one, but all I found were problems with PHP-FPM, which I don't use.

Here's my Nginx configuration file: user nginx; worker_processes 16;

error_log  /var/log/nginx/error.log;

events {
    worker_connections  1024;
}

http {
server_names_hash_max_size 2048;
server_names_hash_bucket_size 512;

server_tokens off;

include    mime.types;
default_type  application/octet-stream;

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout  10;

# Gzip on
gzip on;
gzip_vary on;
gzip_min_length 10240;
gzip_proxied expired no-cache no-store private auth;
gzip_buffers 4 32k;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript text/x-js;
gzip_disable "MSIE [1-6]\.";

# Other configurations
ignore_invalid_headers on;
client_max_body_size    8m;
client_header_timeout  3m;
client_body_timeout 3m;
#send_timeout     3m;
connection_pool_size  256;
client_header_buffer_size 4k;
large_client_header_buffers 4 32k;
request_pool_size  4k;
output_buffers   4 32k;
postpone_output  1460;

# Cache most accessed static files
open_file_cache          max=10000 inactive=10m;
open_file_cache_valid    2m;
open_file_cache_min_uses 1;
open_file_cache_errors   on;

# virtual hosts includes
include "/etc/nginx/conf.d/*.conf";
}

Here's my Nginx vhost file: server { listen ip:80 default_server;

server_name domain.es;
server_name www.domain.es;
server_name ipv4.domain.es;

client_max_body_size 128m;

root "/var/www/vhosts/domain.es/httpdocs";
access_log "/var/www/vhosts/system/domain.es/logs/proxy_access_log";
error_log "/var/www/vhosts/system/domain.es/logs/proxy_error_log";

if ($host ~* ^www.domain.es$) {
    rewrite ^(.*)$ http://domain.es$1 permanent;
}

location / {
    proxy_pass http://82.194.74.41:7080;
    proxy_set_header Host             $host;
    proxy_set_header X-Real-IP        $remote_addr;
    proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
    access_log off;
}

location @fallback {
    proxy_pass http://ip:7080;
    proxy_set_header Host             $host;
    proxy_set_header X-Real-IP        $remote_addr;
    proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
    access_log off;
}

location ~ ^/plesk-stat/ {
    proxy_pass http://ip:7080;
    proxy_set_header Host             $host;
    proxy_set_header X-Real-IP        $remote_addr;
    proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
    access_log off;
}

location ~ ^/(.*\.(ac3|avi|bmp|bz2|css|cue|dat|doc|docx|dts|exe|flv|gif|gz|htm|html|ico|img|iso|jpeg|jpg|js|mkv|mp3|mp4|mpeg|mpg|ogg|pdf|png|ppt|pptx|qt|rar|rm|swf|tar|tgz|txt|wav|xls|xlsx|zip))$ {
    access_log off;
    expires 7d;
    add_header Cache-Control public;

 try_files $uri @fallback;
}

include "/var/www/vhosts/system/domain.es/conf/vhost_nginx.conf";
}

And some of the configuration vars I use with Apache (httpd.conf):

<IfModule prefork.c>
StartServers      14
MinSpareServers    8
MaxSpareServers   14
ServerLimit      1000
MaxClients       1000
MaxRequestsPerChild  2000
</IfModule>

Thank you very much in advance!

Solution 1

In my case, a php extension is missing, after I turned it off, it recovered! Check /var/log/messages to see if there is any segfault.

Solution 2

It seems your Apache is more busy than your Nginx. When Nginx get some requests but Apache can't handle, you get '502 Bad Gateway', which meas Apache refused to work for Nginx.

Try decrease 'worker_connections' and 'worker_processes' in Nginx and increase 'MaxClients', 'ServerLimit'

Make sure worker_connections * worker_processes < MaxClients < ServerLimit

Comments

[Shupyrg]

I reduced the worker_processes to 2 and worker_connections to 512. Also, increased the MaxClients and ServerLimit to 1200. It just happened again, can't even connect to server throught terminal. Thanks for your answer.

[fholzer]

You can't connect to the server because it's probably out of memory. You increased the number of concurrent clients, which means apache will handle more requests concurrently, which results in higher memory usage by php or whatever generates your dynamic content. the only solution imho is to scale up or out. (Increase server memory, or add servers to your farm.)