uwsgi throws IO error caused by uwsgi_response_write_body_do broken pipe

Question

My application is a uwsgi+django setup. I use gevent to do performance testing and run 1200 requests concurrently. At this point, uwsgi will throw an IO error with the following log message:

uwsgi_response_write_body_do(): Broken pipe [core/writer.c line 260]
IOError: write error

Django 1.4.0
uwsgi: 1.9.13
python: 2.6
TCP Listen queue: 1000

What is the cause of this broken pipe error?

Solution 1

This can happen when NGINX started a request to uWSGI but uWSGI took too long to respond, then NGINX closes the connection to uWSGI. When uWSGI finally finishes, it tries to give it's response back to NGINX, but NGINX closed the connection earlier, so then uWSGI throws an I/O error.

So this could mean that your uWSGI process is taking too long.

Update:

uWSGI's Harakiri mode could be useful to automatically terminate such long taking processes if you want to: https://uwsgi-docs.readthedocs.io/en/latest/FAQ.html#what-is-harakiri-mode You might not want to do this because a process might be finishing some long query or something which is necessary.

Solution 2

This error means that the client has closed the connection before uWSGI/Django sends the response. It is generally caused by a timeout in the browser or web server frontend.

To fix it, you need to verify that your setup is correct. Look to see that all of your application's parts (including database adapters) are gevent-friendly. If they're not, you will get no advantage with gevent, and this could even lead to a decrease in performance.

In addition to this, you need to make sure that your database server is able to manage 1200 concurrent connections. If not, it may be ignoring connection attempts.

Solution 3

Now I am not recommending this without you considering your situation. But you could turn uwsgi_ignore_client_abort to "on". With this turned on nginx will keep the aborted connection open until uwsgi returns. Why I don't recommend this completely is because this means an nginx connection will now be tied up until the request finishes. But really the uwsgi thread wasn't being aborted, so aborting the nginx connection early isn't really gaining you much in my opinion.