Pycurl and io.StringIO - pycurl.error: (23, 'Failed writing body)

Question

I'm porting ebay sdk to python3 and I've stumbled upon the following issue.

I'm using pycurl to send some HTTP requests. Here is how I configure it:

    self._curl = pycurl.Curl()
    self._curl.setopt(pycurl.FOLLOWLOCATION, 1)
    self._curl.setopt(pycurl.URL, str(request_url))
    self._curl.setopt(pycurl.SSL_VERIFYPEER, 0)

    self._response_header = io.StringIO()
    self._response_body = io.StringIO()

    self._curl.setopt(pycurl.CONNECTTIMEOUT, self.timeout)
    self._curl.setopt(pycurl.TIMEOUT, self.timeout)

    self._curl.setopt(pycurl.HEADERFUNCTION, self._response_header.write)
    self._curl.setopt(pycurl.WRITEFUNCTION, self._response_body.write)

When I call self._curl.perform() I get the following error:

pycurl.error: (23, 'Failed writing body (1457 != 1460)')

As far as I know this means that there is an issue with the write function, but I can't figure out what it is exactly. Could be related to migration from StringIO module to io, but I'm not sure.

UPD: I've tried the following:

    def body(buf):
        self._response_body.write(buf)

    def header(buf):
        self._response_header.write(buf)

    self._curl.setopt(pycurl.HEADERFUNCTION, header)
    self._curl.setopt(pycurl.WRITEFUNCTION, body)

and it works. I've tried to do the same trick with lambdas (instead of defining those awkward function, but it didn't work.

Solution

I believe the problem is that pycurl no longer functions with StringIO like desired. A solution is to use io.BytesIO instead. You can then get information written into the buffer and decode it into a string.

Using BytesIO with pycurl instead of StringIO:

e = io.BytesIO()
c.setopt(pycurl.WRITEFUNCTION, e.write)

Decoding byte information from the BytesIO object:

htmlString = e.getvalue().decode('UTF-8')

You can use any type of decoding you want, but this should give you a string object you can parse.

Hope this helps people using Python 3.

Comments

[digitalformula]

Coming up on 6 years later and this still helps. Thanks!

[C R]

7 years now and still helping