Python SSL certificate verify error

Question

I'm using requests to access a RESTful API. Everything seems to work. I can authenticate, pull back a session token and even unit test the methods in my class I wrote for the API. Then I tried to run my code.

First, here is the call I'm making. The headers are static session-related items that get set in init(). The body is build dynamically from data in a file and passed in to this function. All of the data is valid.

response = requests.post(url, headers=(Requestheader), data=json.dumps((Requestbody)))

When I run the code, it updates well over 100 records with the metadata I supply. Somewhere around item 150 I get the following:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1045)

My first step was to call the vendor and find out if ALL of their web servers had properly signed certs figuring they were load balancing me and I found a misconfigured server. They tell me this is not the case.

Then I Googled the message and found that there is a verify kwarg, so I tried:

response = requests.post(url, headers=Requestheader, data=json.dumps(Requestbody), verify=False)

I know this isn't ideal long-term, but I wanted to test it to see if The behavior is the same. It did the same thing. It ran for a while and threw the ssl error. I thought the idea of verify=False is that it wouldn't check.

The vendor suggested I check the url I'm using, but it's fine. I would think if there were a proxy server or real man-in-the-middle attack causing problems I wouldn't see success so many times before a failure. I thought maybe it is a session timeout, but that should throw a 401 status and my activity level is too high for an inactivity timeout.

I'm a python noob and not a security professional. Suggestions appreciated.

Solution 1

So the issue might have three resolutions as I see it:

1. A certificate is OK and there is something wrong with the code. The issue may occur, for example, while using prepared requests as described in this solution

   But I don't really think it is your case because in the snippet you've provided no such methods are used. For the two next variants, you'll need to get the URL that causes an error and explore it's certificate (can be done via browser).

2. A certificate is OK but a certificate authority that signed it is not included in CA list that is utilized by requests library. After you'll open a troubling URL, check CA in it and see if it's dates are valid and it is included in this list. If not, add CA in the trusted list for the requests library -- as explained in the answers to this StackOverflow question.

3. A certificate is not valid or self-singed. Same solution as in 2.

The general solution is to wrap your script in the try except clause and to print out all the URLs that will result in mistakes. Then try to request them one by one via requests library and see if the issue occurs. If it does, it's (2) or (3) case. If not — try to run script on another machine with fresh installed python and requests. If the run will be successful — then there's some issue in your configuration.

Solution 2

I have found this over here

I found this solution, insert this code at the beginning of your source file:

import ssl

try:
    _create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
    # Legacy Python that doesn't verify HTTPS certificates by default
    pass
else:
    # Handle target environment that doesn't support HTTPS verification
    ssl._create_default_https_context = _create_unverified_https_context

Solution 3

If you are using Windows and you have already imported the CA in the trusted DB store, you can install the package python-certifi-win32 that automagically will use the same certificates from the Trusted DB store.

pip install python-certifi-win32

Then your code should start to work

See: Python Requests with wincertstore

https://gitlab.com/alelec/python-certifi-win32

Solution 4

For Debian:

Download certificate

openssl s_client -showcerts -connect SERVER:443 </dev/null 2>/dev/null | sed -n -e '/BEGIN\ CERTIFICATE/,/END
CERTIFICATE/ p' > git-mycompany-com.pem

Install

pip install certifi

Execute python

Python 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.

import certifi

certifi.where() '/home/user/detectaEnv/lib/python3.7/site-packages/certifi/cacert.pem'

exit()

Execute command for add perm to python

cat git-mycompany-com.pem | tee -a /home/user/detectaEnv/lib/python3.7/site-packages/certifi/cacert.pem

It has worked perfectly for me

View more solutions