python: [Errno 10054] An existing connection was forcibly closed by the remote host

Question

I am writing python to crawl Twitter space using Twitter-py. I have set the crawler to sleep for a while (2 seconds) between each request to api.twitter.com. However, after some times of running (around 1), when the Twitter's rate limit not exceeded yet, I got this error.

[Errno 10054] An existing connection was forcibly closed by the remote host.

What are possible causes of this problem and how to solve this?

I have searched through and found that the Twitter server itself may force to close the connection due to many requests.

Thank you very much in advance.

Solution 1

This can be caused by the two sides of the connection disagreeing over whether the connection timed out or not during a keepalive. (Your code tries to reused the connection just as the server is closing it because it has been idle for too long.) You should basically just retry the operation over a new connection. (I'm surprised your library doesn't do this automatically.)

Solution 2

there are many causes such as

• The network link between server and client may be temporarily going down.
• running out of system resources.
• sending malformed data.

To examine the problem in detail, you can use Wireshark.

or you can just re-request or re-connect again.

Solution 3

I know this is a very old question but it may be that you need to set the request headers. This solved it for me.

For example 'user-agent', 'accept' etc. here is an example with user-agent:

url = 'your-url-here'
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36'}
r = requests.get(url, headers=headers)

Solution 4

For me this problem arised while trying to connect to the SAP Hana database. When I got this error,

OperationalError: Lost connection to HANA server (ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

I tried to run the code for connection(mentioned below), which created that error, again and it worked.

    import pyhdb
    connection = pyhdb.connect(host="example.com",port=30015,user="user",password="secret")
    cursor = connection.cursor()
    cursor.execute("SELECT 'Hello Python World' FROM DUMMY")
    cursor.fetchone()
    connection.close()

It was because the server refused to connect. It might require you to wait for a while and try again. Try closing the Hana Studio by logging off and then logging in again. Keep running the code for a number of times.

Solution 5

I got the same error ([WinError 10054] An existing connection was forcibly closed by the remote host) with websocket-client after setting ping_interval = 2 in websocket.run_forever(). (I had multiple threads connecting to the same host.)

Setting ping_interval = 10 and ping_timeout = 9 solved the issue. May be you need to reduce the amount of requests and stop making host busy otherwise it will forcibly disconnect you.

View more solutions

Comments

[Roman Hwang]

I have the same problem. Using bottle library, and sending with httplib. I can't really send retry, because the original call was already executed on server. The connection was closed when I tried to read response data. This happens not all the time, usually when I just spam server with requests. Do you know any parameters I can tweak to make the communication stable?

[David Schwartz]

@RomanHwang You either need a way to check on the previous operation without repeating it or you need to make your operations idempotent.

[Roman Hwang]

Thanks for the hint. I also found out the reason of why I get the error so often. It's because of implementation of default development server of bottle. It's single-threaded and is not suited to handle too many requests at a time.

[Don]

Can you add just some details?

[MSIS]

A separate question, please. Any chance you know where Windows10 stores connection strings? I thought it was in C:\Users\User-Name\AppData\Roaming\Microsoft\MicrosoftSQL_Se‌​rver\\110\Tools\Shel‌​l\RegServer.xml (This is for SQL Server, of course)

[Jeremy Thompson]

Extra details: Imagine writing a crawler to poll twitter, and since the crawler isn't a browser it won't have the user-agent by default. So the website is saying please trick us into thinking you're using a real browser with established user-agent settings, like Mozilla, AppleWebKit, Chrome, etc browser.