How to disable hostname checking in requests python

Question

I'm using Requests to connect to a RESTful API. The server I would like to reach use ssl with self-singed certificate.

cafile = "gateway.pem"
r = requests.get(request, auth=('admin', 'password'), verify=cafile)

the problem is I'm getting SSLError of hostname mismatch. there should be a way to disable the hostname checking without disabling certificate validation, as in many java implementations, but I can't find how to do it with requests in python.

stacktrace:

Traceback (most recent call last):
  File "<pyshell#43>", line 1, in <module>
    r = requests.get(request, auth=("admin", "password"), verify='gateway.pem')
  File "C:\Python27\lib\site-packages\requests-2.0.0-py2.7.egg\requests\api.py", line 55, in get
    return request('get', url, **kwargs)
  File "C:\Python27\lib\site-packages\requests-2.0.0-py2.7.egg\requests\api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Python27\lib\site-packages\requests-2.0.0-py2.7.egg\requests\sessions.py", line 357, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python27\lib\site-packages\requests-2.0.0-py2.7.egg\requests\sessions.py", line 460, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python27\lib\site-packages\requests-2.0.0-py2.7.egg\requests\adapters.py", line 358, in send
    raise SSLError(e)
SSLError: hostname '10.76.92.70' doesn't match u'lital.com'

How can this be done?

Solution 1

Requests doesn't allow this directly, however you can provide a custom transport adapter which uses the features of the underlying urllib3. The usage of transport adapters is covered in the requests documentation.

This code is not tested, but should work.

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager


# Never check any hostnames
class HostNameIgnoringAdapter(HTTPAdapter):
    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = PoolManager(num_pools=connections,
                                       maxsize=maxsize,
                                       block=block,
                                       assert_hostname=False)


# Check a custom hostname
class CustomHostNameCheckingAdapter(HTTPAdapter):
    def cert_verify(self, conn, url, verify, cert):
        #      implement me
        host = custom_function_mapping_url_to_hostname(url)
        conn.assert_hostname = host
        return super(CustomHostNameCheckingAdapter,
                     self).cert_verify(conn, url, verify, cert)

In detail the assert_hostname param works as follows: If None use the hostname from the URL, if False suppress hostname checking, if a custom string validate against this string.

Solution 2

I'm a little late to the party but requests_toolbelt looks like it might help if you install version 0.7.0 or newer (my ubuntu 16.04 only has 0.6.0): https://toolbelt.readthedocs.io/en/latest/adapters.html#hostheaderssladapter

From the link:

Example usage:
>>> s.mount('https://', HostHeaderSSLAdapter())
>>> s.get("https://93.184.216.34", headers={"Host": "example.org"})

Solution 3

Did you look into the SSLContext.check_hostname parameter? You should be able to set it to False, and it should not check the hostname:

context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
context.verify_mode = ssl.CERT_REQUIRED
context.check_hostname = True
context.load_default_certs()

The only limitation is that this only works in Python 3.4 and later.

Reference: https://docs.python.org/3/library/ssl.html#ssl.SSLContext.check_hostname

Comments

[user3480498]

When I'm doing the same call with verify=True I'm getting different error of "certificate verify failed". So looks like it is possible to pass the pem file to verify=. when I'm doing the call with cert=cafile instead of verify=cafile, I'm getting SSLError: [Errno 336265225] _ssl.c:351: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib. I added the traceback to the message.

[m.wasowski]

what is your request? Looks like CA is hooked up properly, but it is signed for 'lital.com' and you connect by IP. Hence errors.

[user3480498]

right, the hostname in the certificate ('lital.com') isn't really the server's hostname. That's the reason I want to disable the hostname matching/checking. In java this can be done with ALLOW_ALL_HOSTNAME_VERIFIER. I couldn't find something similar in requests.

[user3480498]

I don't want to disable entirely the certificate validation, only the hostname checking.

[m.wasowski]

OK, now I get it. This is optional client feature as [docs say](rfc6125) and it may be unsupported in request, especially since it's security hazard. You can try: and if you catch this particular error, do manual check and if all seems ok for you, re-request with verify=False. Nothing else comes to my mind.

[m.wasowski]

PS. it doesn't look like it is supported: github.com/kennethreitz/requests/blob/master/requests/… but this may help you: stackoverflow.com/questions/18061640/…

[user3480498]

Thanks for that. I actually tried this as you can see below, but found out that after changing to use my costume adapter, no certificate varification is performed at all (instead of only hostname verification). I passed requests a corrupted pem file and got valid response.

[user3480498]

class MyAdapter(requests.adapters.HTTPAdapter): def init_poolmanager(self, connections, maxsize, block): self._pool_connections = connections self._pool_maxsize = maxsize self._pool_block = block self.poolmanager = requests.packages.urllib3.PoolManager(num_pools=connections, maxsize=maxsize, block=block, assert_hostname=False)

[user3480498]

I'm then doing this: s = requests.Session() s.mount('https://', MyAdapter()) r = s.get(request, auth=("admin", "admin"), verify=cafile)

[user3480498]

Any idea what is worng here and why this disables the certificate validation?

[t-8ch]

If I use a corrupted pem file openssl blows up for me requests.exceptions.SSLError: [Errno 0] _ssl.c:343: error:00000000:lib(0):func(0):reason(0)

[ncoghlan]

The problem with this approach is that it turns off certificate validation completely, it doesn't just turn off the hostname check. In the question, the desired check is for "the presented certificate is signed by the certificate given in cafile", which means that certificate verification needs to remain enabled at the requests level.

[e-info128]

Does not works, the verify argument can not disable hostname validation.