How do I send and receive HTTP POST requests in Python?

Question

I have these two Python scripts I'm using to attempt to work out how to send and receive POST requests in Python:

The Client:

import httplib

conn = httplib.HTTPConnection("localhost:8000")
conn.request("POST", "/testurl")
conn.send("clientdata")
response = conn.getresponse()
conn.close()

print(response.read())

The Server:

from BaseHTTPServer import BaseHTTPRequestHandler,HTTPServer

ADDR = "localhost"
PORT = 8000

class RequestHandler(BaseHTTPRequestHandler):        
    def do_POST(self):
        print(self.path)
        print(self.rfile.read())
        self.send_response(200, "OK")
        self.end_headers()
        self.wfile.write("serverdata")

httpd = HTTPServer((ADDR, PORT), RequestHandler)
httpd.serve_forever()

The problem is that the server hangs on self.rfile.read() until conn.close() has been called on the client but if conn.close() is called on the client the client cannot receive a response from the server. This creates a situation where one can either get a response from the server or read the POST data but never both. I assume there is something I'm missing here that will fix this problem.

Additional information:

conn.getresponse() causes the client to hang until the response is received from the server. The response doesn't appear to be received until the function on the server has finished execution.

Solution 1

There are a couple of issues with your original example. The first is that if you use the request method, you should include the message body you want to send in that call, rather than calling send separately. The documentation notes send() can be used as an alternative to request:

As an alternative to using the request() method described above, you can also send your request step by step, by using the four functions below.

You just want conn.request("POST", "/testurl", "clientdata").

The second issue is the way you're trying to read what's sent to the server. self.rfile.read() attempts to read the entire input stream coming from the client, which means it will block until the stream is closed. The stream won't be closed until connection is closed. What you want to do is read exactly how many bytes were sent from the client, and that's it. How do you know how many bytes that is? The headers, of course:

length = int(self.headers['Content-length'])
print(self.rfile.read(length))

I do highly recommend the python-requests library if you're going to do more than very basic tests. I also recommend using a better HTTP framework/server than BaseHTTPServer for more than very basic tests (flask, bottle, tornado, etc.).

Solution 2

Long time answered but came up during a search so I bring another piece of answer. To prevent the server to keep the stream open (resulting in the response never being sent), you should use self.rfile.read1() instead of self.rfile.read()