nginx connection timeout & client closed connection issue

Question

I have this nginx server running on AWS & it was working all fine until recently when couple of users started complaining about the website not opening until they made some 10 attempts to access it.

I never was able to repro the issue from my side. I am using google's dns i.e 8.8.8.8 & when I changed the same for one of the users, the site was working fine. Now this can be the reason or this can be just a coincidence also.

I found this in the error log -

2014/05/29 13:46:15 [info] 6940#0: *150649 client timed out (110: Connection timed out) while waiting for request, client: xx.xxx.xxx.xx, server: 0.0.0.0:80
2014/05/29 13:46:20 [info] 6940#0: *150670 client closed connection while waiting for request, client: xx.xxx.xxx.xx, server: 0.0.0.0:80
2014/05/29 13:46:20 [info] 6940#0: *150653 client closed connection while waiting for request, client: xx.xxx.xxx.xx, server: 0.0.0.0:80
2014/05/29 13:46:20 [info] 6940#0: *150652 client closed connection while waiting for request, client: xx.xxx.xxx.xx, server: 0.0.0.0:80

And some places even this -

2014/05/29 13:46:53 [info] 6940#0: *150665 client closed connection while waiting for request, client: xx.xxx.xxx.xx, server: 0.0.0.0:80
2014/05/29 13:46:53 [info] 6940#0: *150660 client xx.xxx.xxx.xx closed keepalive connection

Note- Have placed xx.xxx.xxx.xx for the clien't IP

Here is the nginx config -

server {
    listen       80;
    server_name  somedomain.com  www.somedomain.com;

    #charset koi8-r;
    #access_log  /var/log/nginx/log/host.access.log  main;

    root        /var/www/somedomain/current/app/webroot;
    index       index.php index.html index.htm;

    ... couple of location rules ...
}

I would really appreciate any help.

Thanks

Solution

Based on the log you provided from Nginx, it seems that the connections between your server and users are unstable or slow. Please try traceroute to your client IP address or his/her gateway from your server. Also, ping your client IP address for a long time to see the packet loss rate and response time. MTU may be another source of this problem. Test if you can reach your client with MTU=1500 (Mac: ping -D -s 1472 xx.xx.xx.xx).

BTW: If your server or client resides in China, this problem usually not your fault. GFW is known to randomly discard packets between border to intentionally make international connection quality worse.

Comments

[Andrew S]

This could be a problem with the developers' connection to the server, not the server. Since you cannot recreate the problem and the server itself is registering a client connection timeout, we need to suspect the developer may be behind a firewall and they have internal networking issues that cause this.

[Alfonso]

You can try disabling Keep-Alive just as a test for this issue. I'm not sure the traffic hitting your webserver but Keep-Alive could be causing you to hit the concurrency limit in your nginx config. Here is more info: nginx.com/blog/http-keepalives-and-web-performance

[Ethan Collins]

@NitishDhar Did you get to solve this problem? I am also facing the same issue and just clueless. Will be glad if you can share the solution.

[GeorgeB]

Questions: is the server behind a load balancer or a firewall? Is NAT involved? Is there a tunnel of any sort between the server and the Internet? The reason I ask is that this sounds like the sort of thing that happens when there is a tunnel someplace in the path and someone has blocked all ICMP which breaks Path MTU discovery.

[Roshan]

fyi, GFW = Great Firewall of China.