Python HTTPS requests (urllib2) to some sites fail on Ubuntu 12.04 without proxy

Question

I have an little app I wrote in Python and it used to work... until yesterday, when it suddenly started giving me an error in a HTTPS connection. I don't remember if there was an update, but both Python 2.7.3rc2 and Python 3.2 are failing just the same.

I googled it and found out that this happens when people are behind a proxy, but I'm not (and nothing have changed in my network since the last time it worked). My syster's computer running windows and Python 2.7.2 has no problems (in the same network).

>>> url = 'https://www.mediafire.com/api/user/get_session_token.php'
>>> response = urllib2.urlopen(url).read()
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 400, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 418, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1215, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1177, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>

What's wrong? Any help is appreciated.

PS.: Older python versions don't work either, not in my system and not in a live session from USB, but DO work in a Ubuntu 11.10 live session.

Solution 1

This appears to be related to the addition of TLS 1.1 and 1.2 support to the version of OpenSSL found in 12.04. The connection failure can be reproduced with the OpenSSL command line tool:

$ openssl s_client -connect www.mediafire.com:443
CONNECTED(00000003)
140491065808544:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 320 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

The connection succeeds if I force the connection to use TLS 1.0 with the -tls1 command line argument.

I would suggest you file a bug report about this problem here:

https://bugs.launchpad.net/ubuntu/+filebug

Solution 2

For python novices like me, here is the way to override httplib the easiest way. At the top of your python script, include these lines:

import httplib
from httplib import HTTPConnection, HTTPS_PORT
import ssl

class HTTPSConnection(HTTPConnection):
    "This class allows communication via SSL."
    default_port = HTTPS_PORT

    def __init__(self, host, port=None, key_file=None, cert_file=None,
            strict=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
            source_address=None):
        HTTPConnection.__init__(self, host, port, strict, timeout,
                source_address)
        self.key_file = key_file
        self.cert_file = cert_file

    def connect(self):
        "Connect to a host on a given (SSL) port."
        sock = socket.create_connection((self.host, self.port),
                self.timeout, self.source_address)
        if self._tunnel_host:
            self.sock = sock
            self._tunnel()
        # this is the only line we modified from the httplib.py file
        # we added the ssl_version variable
        self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file, ssl_version=ssl.PROTOCOL_TLSv1)

#now we override the one in httplib
httplib.HTTPSConnection = HTTPSConnection
# ssl_version corrections are done

From here on, you can use urllib or whatever you use just like you normally would.

Note: This is for python 2.7. For a python 3.x solution, you need to override the HTTPSConnection class found in http.client. I leave that as an exercise for the reader. :-)

Solution 3

You can avoid modifying the httplib.py file by modifying your HTTPSConnection object:

import httplib, ssl, socket

conn = httplib.HTTPSConnection(URL.hostname)
sock = socket.create_connection((conn.host, conn.port), conn.timeout, conn.source_address)
conn.sock = ssl.wrap_socket(sock, conn.key_file, conn.cert_file, ssl_version=ssl.PROTOCOL_TLSv1)
conn.request('POST', URL.path + URL.query)

The request method creates a new socket only if connection.sock is not defined. Creating your own one adding the ssl_version parameter will make the request method use it. Then everything else works as usual.

I was having the same issue and this works for me.

Regards

Solution 4

The problem is in ssl, it has nothing to do with HTTP, so why patching httplib if you can patch ssl. The following code should fix all SSL sockets including, but not limited to HTTPS, for Python 2.6+ (built in ssl, did not try with pyopenssl).

import functools
import ssl

old_init = ssl.SSLSocket.__init__

@functools.wraps(old_init)
def ubuntu_openssl_bug_965371(self, *args, **kwargs):
  kwargs['ssl_version'] = ssl.PROTOCOL_TLSv1
  old_init(self, *args, **kwargs)

ssl.SSLSocket.__init__ = ubuntu_openssl_bug_965371

Solution 5

EDIT httplib.py (/usr/lib/pythonX.X/httplib.py on Linux)

FIND HTTPSConnection class declaration

  class HTTPSConnection(HTTPConnection):
....

Inside class code CHANGE line

self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file)

TO

self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file, ssl_version=ssl.PROTOCOL_TLSv1)

Then httplib HTTPS request should work

import httplib
from urlparse import urlparse
url = XXX
URL = urlparse(url)
connection = httplib.HTTPSConnection(URL.hostname)
connection.request('POST', URL.path + URL.query)
response = connection.getresponse()

View more solutions

Comments

[Admin]

Does it happen for every SSL site you try to contact, or just the one? If it doesn't occur for every site, then could you tell us what site is causing the problem?

[Admin]

Oh, I forgot to mention: the site is Mediafire. It's its get_session_token call that is causing the problem.

[Admin]

This happens with stream.twitter.com for me at the time of writing.

[Pablo]

Thank you! I reported a bug. Please, see if you can add any relevant info to it: bugs.launchpad.net/ubuntu/+source/openssl/+bug/965371

[nanofarad]

It really isn;t right to edit a system file like that. Instead, redefine any definitions that need to be changed, by redefining them in your code.

[MarkR]

I really like this solution, it avoids modifying any system libraries or other hackery.

[Ben Walther]

Fails using Python 2.7.4 on Ubuntu 12.04: NameError: name 'socket' is not defined. --- You'll need to add "import socket" as well.

[dharmatech]

Works great on Ubuntu 13.04. Thanks!

[Cerin]

How does this help him work around the problem in Python?

[Cerin]

This gives me the error BadStatusLine: ''

[sureshvv]

TypeError: unbound method __init__() must be called with SSLSocket instance as first argument (got _socketobject instance instead)

[chnrxn]

Hmm, partial() doesn't work for class methods. Will post a better solution shortly.

[chnrxn]

It's hackish, but it works rather well in today's context. Ever since the poodle vulnerability has been discovered, TLSv1 pretty much became the only acceptable version on the Internet.

[chnrxn]

@sureshvv, if you can help to check the solution it will be appreciated.

[chnrxn]

Good answer. Nice, elegant way to solve the problem.

[sureshvv]

@temeto's answer worked.