How to handle a broken pipe (SIGPIPE) in python?

Question

I've written a simple multi-threaded game server in python that creates a new thread for each client connection. I'm finding that every now and then, the server will crash because of a broken-pipe/SIGPIPE error. I'm pretty sure it is happening when the program tries to send a response back to a client that is no longer present.

What is a good way to deal with this? My preferred resolution would simply close the server-side connection to the client and move on, rather than exit the entire program.

PS: This question/answer deals with the problem in a generic way; how specifically should I solve it?

Solution 1

Read up on the try: statement.

try:
    # do something
except socket.error, e:
    # A socket error
except IOError, e:
    if e.errno == errno.EPIPE:
        # EPIPE error
    else:
        # Other error

Solution 2

Assuming that you are using the standard socket module, you should be catching the socket.error: (32, 'Broken pipe') exception (not IOError as others have suggested). This will be raised in the case that you've described, i.e. sending/writing to a socket for which the remote side has disconnected.

import socket, errno, time

# setup socket to listen for incoming connections
s = socket.socket()
s.bind(('localhost', 1234))
s.listen(1)
remote, address = s.accept()

print "Got connection from: ", address

while 1:
    try:
        remote.send("message to peer\n")
        time.sleep(1)
    except socket.error, e:
        if isinstance(e.args, tuple):
            print "errno is %d" % e[0]
            if e[0] == errno.EPIPE:
               # remote peer disconnected
               print "Detected remote disconnect"
            else:
               # determine and handle different error
               pass
        else:
            print "socket error ", e
        remote.close()
        break
    except IOError, e:
        # Hmmm, Can IOError actually be raised by the socket module?
        print "Got IOError: ", e
        break

Note that this exception will not always be raised on the first write to a closed socket - more usually the second write (unless the number of bytes written in the first write is larger than the socket's buffer size). You need to keep this in mind in case your application thinks that the remote end received the data from the first write when it may have already disconnected.

You can reduce the incidence (but not entirely eliminate) of this by using select.select() (or poll). Check for data ready to read from the peer before attempting a write. If select reports that there is data available to read from the peer socket, read it using socket.recv(). If this returns an empty string, the remote peer has closed the connection. Because there is still a race condition here, you'll still need to catch and handle the exception.

Twisted is great for this sort of thing, however, it sounds like you've already written a fair bit of code.

Solution 3

SIGPIPE (although I think maybe you mean EPIPE?) occurs on sockets when you shut down a socket and then send data to it. The simple solution is not to shut the socket down before trying to send it data. This can also happen on pipes, but it doesn't sound like that's what you're experiencing, since it's a network server.

You can also just apply the band-aid of catching the exception in some top-level handler in each thread.

Of course, if you used Twisted rather than spawning a new thread for each client connection, you probably wouldn't have this problem. It's really hard (maybe impossible, depending on your application) to get the ordering of close and write operations correct if multiple threads are dealing with the same I/O channel.

Comments

[Adam Plumb]

If I do a try: #something except: #anything, will it just catch anything, and not just IOErrors?

[user1066101]

The blanket except is a bad policy. But yet, it will catch any kind of exception. You know it's an IOError. Handle that. If something else crops up, figure out why and handle it appropriately. You don't want to mask bugs like division by zero or out of memory.

[mhawke]

If you are using Python's socket module, you will not get an IOError exception: you will get a socket.error exception.

[mhawke]

The errno would be 32, not 23.

[mhawke]

You are not going to get an IOError. socket.error does not have the errno attribute for broken pipe - this code will raise AttributeError.

[mhawke]

You won't get IOError with errno == EPIPE for broken pipe socket exceptions, you will get socket.error, so there's no point in checking for it in the IOError exception handler. You have 2 votes for a (still) bad answer. Perhaps you should up vote my answer :)

[user1066101]

At this point, the questioner should know how to use the try: statement.

[mhawke]

Agree that the questioner should now have some clue as to what to do. You code snippet is still incorrect though, and the questioner might follow your example. It won't break his code, it's just that checking for EPIPE in the IOError handler is useless.

[user1066101]

@mhawke: you're still right. Both times. However, it's hard to contrive an example of standard OS errors (with errno) and other errors (without errno). I think it's important to have a tidy code sample -- I'm not writing their app for them.

[Kirk Strauser]

I should have clarified that I meant "23" as a placeholder. Really? 32? I was closer than I would have guessed. :-)

[trevorKirkby]

Blanket excepts may usually be a bad policy but in this case it is really not. This is more or less what exceptions are for. It is almost always better to use if statements to test that your given inputs and calls won't throw errors, because that won't screw up your debugging. However sockets have lower level resources then can be easily accessed with the python socket module. Since we can't really access the pipe and handle it appropriately on the lower level, exceptions are the way to go.

[Piotr Dobrogost]

The simple solution is not to shut the socket down before trying to send it data. You assume here that the socket was shut down locally (on the server side) whereas in this answer we read that This usually happens when you write to a socket fully closed on the other (client) side. Did you omit this case by purpose or you don't agree with this statement?

[guettli]

This looks strange if isinstance(e.args, tuple):. Can someone explain this?

[mjz19910]

That means, "is e.args a tuple?"

[tartaruga_casco_mole]

Why is EPIPE not always raised on first write? What is the condition that EPIPE is raised?