urllib and "SSL: CERTIFICATE_VERIFY_FAILED" Error

python python-2.7 ssl ssl-certificate urllib

876,746

Solution 1

If you just want to bypass verification, you can create a new SSLContext. By default newly created contexts use CERT_NONE.

Be careful with this as stated in section 17.3.7.2.1

When calling the SSLContext constructor directly, CERT_NONE is the default. Since it does not authenticate the other peer, it can be insecure, especially in client mode where most of time you would like to ensure the authenticity of the server you’re talking to. Therefore, when in client mode, it is highly recommended to use CERT_REQUIRED.

But if you just want it to work now for some other reason you can do the following, you'll have to import ssl as well:

input = input.replace("!web ", "")      
url = "https://domainsearch.p.mashape.com/index.php?name=" + input
req = urllib2.Request(url, headers={ 'X-Mashape-Key': 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' })
gcontext = ssl.SSLContext()  # Only for gangstars
info = urllib2.urlopen(req, context=gcontext).read()
Message.Chat.SendMessage ("" + info)

This should get round your problem but you're not really solving any of the issues, but you won't see the [SSL: CERTIFICATE_VERIFY_FAILED] because you now aren't verifying the cert!

To add to the above, if you want to know more about why you are seeing these issues you will want to have a look at PEP 476.

This PEP proposes to enable verification of X509 certificate signatures, as well as hostname verification for Python's HTTP clients by default, subject to opt-out on a per-call basis. This change would be applied to Python 2.7, Python 3.4, and Python 3.5.

There is an advised opt out which isn't dissimilar to my advice above:

import ssl

# This restores the same behavior as before.
context = ssl._create_unverified_context()
urllib.urlopen("https://no-valid-cert", context=context)

It also features a highly discouraged option via monkeypatching which you don't often see in python:

import ssl

ssl._create_default_https_context = ssl._create_unverified_context

Which overrides the default function for context creation with the function to create an unverified context.

Please note with this as stated in the PEP:

This guidance is aimed primarily at system administrators that wish to adopt newer versions of Python that implement this PEP in legacy environments that do not yet support certificate verification on HTTPS connections. For example, an administrator may opt out by adding the monkeypatch above to sitecustomize.py in their Standard Operating Environment for Python. Applications and libraries SHOULD NOT be making this change process wide (except perhaps in response to a system administrator controlled configuration setting).

If you want to read a paper on why not validating certs is bad in software you can find it here!

Solution 2

This isn't a solution to your specific problem, but I'm putting it here because this thread is the top Google result for "SSL: CERTIFICATE_VERIFY_FAILED", and it lead me on a wild goose chase.

If you have installed Python 3.6 on OSX and are getting the "SSL: CERTIFICATE_VERIFY_FAILED" error when trying to connect to an https:// site, it's probably because Python 3.6 on OSX has no certificates at all, and can't validate any SSL connections. This is a change for 3.6 on OSX, and requires a post-install step, which installs the certifi package of certificates. This is documented in the file ReadMe.rtf, which you can find at /Applications/Python\ 3.6/ReadMe.rtf (see also the file Conclusion.rtf, and the script build-installer.py that generates the macOS installer).

The ReadMe will have you run the post-install script at /Applications/Python\ 3.6/Install\ Certificates.command (its source is install_certificates.command), which:

first installs the Python package certifi, and
then creates a symbolic link from the OpenSSL certificates file to the certificates file installed by the package certifi.

Release notes have some more info: https://www.python.org/downloads/release/python-360/

On newer versions of Python, there is more documentation about this:

Solution 3

To expand on Craig Glennie's answer:

in Python 3.6.1 on MacOs Sierra

Entering this in the bash terminal solved the problem:

pip install certifi
/Applications/Python\ 3.6/Install\ Certificates.command

Solution 4

On Windows, Python does not look at the system certificate, it uses its own located at ?\lib\site-packages\certifi\cacert.pem.

The solution to your problem:

download the domain validation certificate as *.crt or *pem file
open the file in editor and copy it's content to clipboard
find your cacert.pem location: from requests.utils import DEFAULT_CA_BUNDLE_PATH; print(DEFAULT_CA_BUNDLE_PATH)
edit the cacert.pem file and paste your domain validation certificate at the end of the file.
Save the file and enjoy requests!

Solution 5

I was having a similar problem, though I was using urllib.request.urlopen in Python 3.4, 3.5, and 3.6. (This is a portion of the Python 3 equivalent of urllib2, per the note at the head of Python 2's urllib2 documentation page.)

My solution was to pip install certifi to install certifi, which has:

... a carefully curated collection of Root Certificates for validating the trustworthiness of SSL certificates while verifying the identity of TLS hosts.

Then, in my code where I previously just had:

import urllib.request as urlrq

resp = urlrq.urlopen('https://example.com/bar/baz.html')

I revised it to:

import urllib.request as urlrq
import certifi

resp = urlrq.urlopen('https://example.com/bar/baz.html', cafile=certifi.where())

If I read the urllib2.urlopen documentation correctly, it also has a cafile argument. So, urllib2.urlopen([...], certifi.where()) might work for Python 2.7 as well.

UPDATE (2020-01-01): As of Python 3.6, the cafile argument to urlopen has been deprecated, with the context argument supposed to be specified instead. I found the following to work equally well on 3.5 through 3.8:

import urllib.request as urlrq
import certifi
import ssl

resp = urlrq.urlopen('https://example.com/bar/baz.html', context=ssl.create_default_context(cafile=certifi.where()))

View more solutions

876,746

Author by

user3724476

Updated on January 05, 2022

Comments

user3724476 over 2 years

I am getting the following error:

Exception in thread Thread-3:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in        __bootstrap_inner
self.run()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in  run
self.__target(*self.__args, **self.__kwargs)
File "/Users/Matthew/Desktop/Skypebot 2.0/bot.py", line 271, in process
info = urllib2.urlopen(req).read()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1240, in https_open
context=self._context)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1197, in do_open
raise URLError(err)
URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>

This is the code that is causing this error:

if input.startswith("!web"):
    input = input.replace("!web ", "")      
    url = "https://domainsearch.p.mashape.com/index.php?name=" + input
    req = urllib2.Request(url, headers={ 'X-Mashape-Key': 'XXXXXXXXXXXXXXXXXXXX' })
    info = urllib2.urlopen(req).read()
    Message.Chat.SendMessage ("" + info)

The API I'm using requires me to use HTTPS. How can I make it bypass the verification?

pyCthon about 9 years

So if this error is keeping me from using setup.py upload how can I fix that?
ihightower about 9 years

ssl._create_default_https_context = ssl._create_unverified_context this worked for me as mentioned by Noelkd near the end. As ours is an intranet site for HP printers... purely used for scraping... we don't have an issue with using this method.
Kevin Smyth over 8 years

lib\site-packages\certifi\cacert.pem does not exist in Python 2.7.10. And the question is about urllib2 not requests
digz6666 over 8 years

Best solution so far, twitter API uses DigiCert certificate which is not in my python's cacert.pem file. I added there and VOILA!
marcadian almost 8 years

I encountered it on 2.7.12 on mac, using urllib2 library, using requets library seems fine though
trevorKirkby over 7 years

I suspect that disabling all HTTP verification for all of python is a bit excessive to deal with one error in verification. There are a lot of cases in which one might want verification by default.
muyong about 7 years

Directly run /Applications/Python\ 3.6/Install\ Certificates.command solve my problem on OSX. Thanks.
Bill Bell about 7 years

I'm writing as a reviewer. Would you please add some words of explanation for your answer. Otherwise, your answer may be subject to deletion.
ofavre about 7 years

Worked on Debian. For the records, the file path on my system was: /usr/local/lib/python2.7/dist-packages/certifi-2015.09.06.2-‌py2.7.egg/certifi/ca‌cert.pem. Thanks!
y2knoproblem about 7 years

I'm using OS X El Capitan 10.11.6, Python 3.6.0 and this solution worked perfectly.
Christopher Thorjussen about 7 years

Worked for me on Windows Server 2016 on python 2.7.13. I normally don't have python installed on windows servers but needed it for a vCenter upgrade with Nexus 1000v migration to VDS, and this just fixes the problem with a self-signed vCenter cert for the time being (will use VMCA and valid certs in upgraded environment). I had no luck getting the cert to be accepted by editing the cacert.pem in python request package
Christopher Thorjussen about 7 years

Also, I set the variable directly in command line window before running the script, so it won't be disabling all checks like @someone-or-other are afraid of, which I too wouldn't recommend.
jww almost 7 years

You first method is the least desirable one - bypass validation. Why is the one that actually validates the certificate not listed first?
sorin almost 7 years

That's perfect for my use case: testing. I would never do this on production, but for running tests which have nothing to do with SSL, that's a wonderful option.
Mahdi Yusuf almost 7 years

this can also happen if you have an outdate version of openSSL. So for me what worked was updating to most recent version of certifi and updating openssl on the boxes.
James almost 7 years

I'm using macOS Sierra 10.12.5, Python 3.6.1 and this solution worked perfectly.
Ian over 6 years

i had to run this even though i was running a virtualenv with certifi installed.... worked though.
prayagupa over 6 years

context is what I needed
hBy2Py over 6 years

@marcadian Installing and using certifi, which is apparently a scrape of the certificates held by requests, fixed the problem for me on Python 3.4 to 3.6.
zombi_man over 6 years

Thank you Craig! I had to install certifi manually via pip3. After that your command worked for me.
tommy.carstensen over 6 years

@CraigGlennie How do I solve the problem, if I don't have admin rights? Thanks!
jww over 6 years

He stated he is on OS X, not vCenter 6.
Craig Glennie about 6 years

@tommy.carstensen you may need to try using a virtualenv, and installing the certifi package in there.
tommy.carstensen about 6 years

@CraigGlennie I did brew install in the end of Python and SSL certificates.
bmosov01 about 6 years

I'm under the gun to get out some analysis, and your post saved me losing hours chasing down a red herring. Much appreciated.
Davos about 6 years

Don't hang your head in shame for forgetting something and then figuring it out and sharing, someone might find that useful. Using IE though... shame :p
None about 6 years

Can you help me plz solve this relevant problem in python and ssl validation link
Guilherme Iazzetta about 6 years

This work PERFECTLY in my Macbook. BUT, I have the same error in AWS. I can't run this command in my ec2.
jamshid almost 6 years

Unfortunately python requests does not use any operating system's CA trust store. github.com/requests/requests/issues/2966. You have to set REQUESTS_CA_BUNDLE github.com/bloomreach/s4cmd/issues/111#issuecomment-40683951‌4.
mbello over 5 years

Thanks, this solution did it for me: import os os.environ["PYTHONHTTPSVERIFY"] = "0"
rapport89 over 5 years

tested this solution in Jenkins environment, personal environment, this works!
natonomo over 5 years

Verified this also works on Python 3.7. Thanks, this is definitely the quickest and simplest solution!
KingKongCoder over 5 years

try sudo /Applications/Python\ 3.6/Install\ Certificates.command if permissions are denied.
wolfog over 5 years

thanks to @Noelkd ,this works for me . /** import ssl context = ssl._create_unverified_context() urllib.urlopen("no-valid-cert", context=context) */
tblznbits over 5 years

Works on Windows 10, Python 3.6, with the Google and Twitter APIs and the requests module in general.
hpm over 5 years

Verified that this does not in fact work on Python 3.6. The direct command, pip install certifi, does not fix the issue either. -1
hpm over 5 years

This does not in fact fix the issue on Python 3.6. -1
Erdős-Bacon over 5 years

If using the command line freaks you out (probably not if you're here, but hey), you can also find the Applications folder (or equivalent on other OSes) that Python created on installation, then open the script by double-clicking. Specific name of the folder depends on the version of Python you installed.
dragon788 over 5 years

This is extremely unsafe as it bypasses ALL certificate verification, use at your own risk and for the love of ANYBODY don't use this in production code.
JayRizzo over 5 years

This fixed it for me /Applications/Python\ 3.7/Install\ Certificates.command running this directly in terminal! Thank you @CraigGlennie & @muyong I appreciate the out-of-the-box-thinking for placing this here!
user124384 over 5 years

I didn't use the Python installer when I installed Python (maybe I used homebrew?), so I couldn't find where this command was. Per the suggestion here stackoverflow.com/a/49953648/3322074, I reinstalled Python 3 with the installer, and the directions for running this script were in the installation complete window.
Biranchi over 5 years

Thanks. Finally it worked. It's not necessary that Python 3.6 is installed under /Applications on Mac OS. This should be the accepted answer.
Asclepius over 5 years

You should never ever do such monkeypatching. It is extremely dangerous and will affect all downstream use of the code.
Iqbal over 5 years

May be stupid question, but what is CA.crt and where can I find it?
Ganesh Chowdhary Sadanala about 5 years

see this video it helped me alot to understand what ssl is. video URL:youtu.be/dsuVPxuU_hc
Roman about 5 years

Still doesn't help with Python3.7 and macOS 10.14.4
Dan Walters about 5 years

Anyone got this to work on Debian or equivalent? Tried installing python-certifi but didn't work.
klbytec about 5 years

Worked like a charm for Python 3.7.3. Thank you.
Norman Breau about 5 years

You're answer led me onto the right track for me. On ubuntu, I had to set SSL_CERT_DIR to /etc/ssl/certs, and also made sure the ca-certificates package was installed and up to date.
Manu mathew about 5 years

Worked like charm in Mac OS X.
Jimmy Li about 5 years

I can't find /Application/Python 3.6/ directory. I installed Anaconda. Can anyone tell me how to do it with python installed with Anaconda?
Thomas almost 5 years

Python 3.7 on macOS Mojave: it doesn't work here. I've uninstalled / reinstalled / run the install certificate, rebooted, etc.. doesn't work for me
James Parker over 4 years

How does this work if you are using pyenv along with virtualenv? When I go to ~/.pyenv/versions/3.6.0/ I don't see an Install directory.
xergiopd over 4 years

worked for me with Python3.7 on macOS Mojave, I was using virtualenv so I removed the virtual dir, install the certificate and then recreate the virtualenv.
Amin Hemati Nik over 4 years

thanks ,solved the problem for OS X 10.11.6 and python 3.7
milanDD over 4 years

After using ssl._create_default_https_context = ssl._create_unverified_context, how do I set it back to the default behavior? (verified context)
demongolem over 4 years

-bash: /Applications/Python 3.6/Install Certificates.command: No such file or directory
ostrokach over 4 years

load_verify_locations mutates the SSLContext instance and returns None. You should use context=ssl.create_default_context(cafile=certifi.where()) instead. See the ssl docs for more info.
hBy2Py over 4 years

@ostrokach Huh, I tried something like that, but I must've used the wrong ssl function. See edit; ok?
numbermaniac over 4 years

I installed Python via Homebrew so I don't have this .command file. What's the solution then?
Josh about 4 years

This worked for me on macOS Catalina 10.15.2, brew installed Python 3.7.6, with a self-signed certificate which I exported from Keychain as a .pem. I had already trusted the certificate in Keychain, but apparently Python does not look at that. Thanks!
Joseph Astrahan about 4 years

For 3.8 do this, /Applications/Python\ 3.8/Install\ Certificates.command
Nischay Namdev about 4 years

Thanks. I literally searched for 100's of solutions and yours worked.
ascourtas about 4 years

can you just pip install certifi instead, or is that insufficient?
Kunal almost 4 years

In the words of an old colleague... "you are a champion man"
Alexey Burdin over 3 years

This is the answer. Thanks.
0plus1 over 3 years

This is the correct solution.. but.. how do you deal with application "compiled" with py2app and similar?
mircealungu over 3 years

Tried several other solutions: this worked on 10.14 Mojave
Dave Davis over 3 years

This seemed too simple and I passed this by on my initial search. I'm kicking myself now. It's pretty confusing on Ubuntu why Python's Certifi uses separate bundled certificate bundles instead of defaulting to the system ones first. Thanks for saving my week.
Renaud over 3 years

you're my hero!
Dounchan over 3 years

Great! Worked for me!...Actually, for my website, I had to concatenate both cert and chain files to cacert.pem.
Frank over 3 years

Tested also work on windows, change the export to set.
donut about 3 years

I am using MacOS Catalina and Python 3.7. Directly run /Applications/Python\ 3.7/Install\ Certificates.command solve my problem on OSX. Thanks.
happy-integer about 3 years

I was using Python 2.7 after updating Python to 2.7.17 worked like a charm, thank you so much.
Ybz almost 3 years

this comment is gold, do not overlook it like i first did, this solved the problem for me
Henry Navarro almost 3 years

I had the same problem in Ubuntu I following your instructions I could solve it. Thank you!
C-Y over 2 years

For me I have to use gcontext = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
Swapnil over 2 years

This is helpful in resolving issue on Ubuntu machine
praba230890 over 2 years

when you face SSL cert error when running flask run create-app using flask-appbuilder in Mac, this will help you without the need to run sudo
tash over 2 years

I just wanted to note that after this line in boto package, I had to add ssl._create_default_https_context = ssl._create_unverified_context .
BlueDogRanch about 2 years

@hBy2Py I'm thinking this may fix my issue askubuntu.com/questions/1401379/certificate-verify-failed-er‌ror Question: what is baz.html? The template URL of your Django app? And do the imports go into views.py?
BlueDogRanch about 2 years

Hi @veganaiZe Why does the haxx.se cert work? Is this something I should try for askubuntu.com/questions/1401379/certificate-verify-failed-er‌ror
veganaiZe about 2 years

Hi @BlueDogRanch, Not sure exactly what you mean; Maybe this link helps? curl.se/docs/caextract.html
BlueDogRanch about 2 years

Thanks, that helped with an explanation, but using it didn't fix my issue.
veganaiZe about 2 years

Well then you might want to create a new post with your issue @BlueDogRanch.
hBy2Py about 2 years

To answer your specific questions here, @BlueDogRanch: 1) https://.../baz.html is just a fake website address, to demonstrate the syntax. 2) The imports need to go in whatever module where the urlopen() (or whatever resource opener) function is called, and the new SSL context pointing to the certifi certificates needs to be passed into the opener as the context keyword argument).
Amin.A about 2 years

This worked for me, CentOS Linux release 7.9.2009. I did use export SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt instead. As Charlie Burns pointed out, this issue is already discussed in detail in access.redhat.com/articles/2039753
matbrgz about 2 years

For MacOS 12.4 and Python 3.6 I had to run the install certificates command as root. Thanks!
TazExprez almost 2 years

Thanks a lot for this! I just went to the Applications Folder, found the Python folder, and clicked on the Install Certificates.command file.