Correct way to try/except using Python requests module?

Question

try:
    r = requests.get(url, params={'s': thing})
except requests.ConnectionError, e:
    print e #should I also sys.exit(1) after this?

Is this correct? Is there a better way to structure this? Will this cover all my bases?

Solution 1

Have a look at the Requests exception docs. In short:

In the event of a network problem (e.g. DNS failure, refused connection, etc), Requests will raise a ConnectionError exception.

In the event of the rare invalid HTTP response, Requests will raise an HTTPError exception.

If a request times out, a Timeout exception is raised.

If a request exceeds the configured number of maximum redirections, a TooManyRedirects exception is raised.

All exceptions that Requests explicitly raises inherit from requests.exceptions.RequestException.

To answer your question, what you show will not cover all of your bases. You'll only catch connection-related errors, not ones that time out.

What to do when you catch the exception is really up to the design of your script/program. Is it acceptable to exit? Can you go on and try again? If the error is catastrophic and you can't go on, then yes, you may abort your program by raising SystemExit (a nice way to both print an error and call sys.exit).

You can either catch the base-class exception, which will handle all cases:

try:
    r = requests.get(url, params={'s': thing})
except requests.exceptions.RequestException as e:  # This is the correct syntax
    raise SystemExit(e)

Or you can catch them separately and do different things.

try:
    r = requests.get(url, params={'s': thing})
except requests.exceptions.Timeout:
    # Maybe set up for a retry, or continue in a retry loop
except requests.exceptions.TooManyRedirects:
    # Tell the user their URL was bad and try a different one
except requests.exceptions.RequestException as e:
    # catastrophic error. bail.
    raise SystemExit(e)

---

As Christian pointed out:

If you want http errors (e.g. 401 Unauthorized) to raise exceptions, you can call Response.raise_for_status. That will raise an HTTPError, if the response was an http error.

An example:

try:
    r = requests.get('http://www.google.com/nothere')
    r.raise_for_status()
except requests.exceptions.HTTPError as err:
    raise SystemExit(err)

Will print:

404 Client Error: Not Found for url: http://www.google.com/nothere

Solution 2

One additional suggestion to be explicit. It seems best to go from specific to general down the stack of errors to get the desired error to be caught, so the specific ones don't get masked by the general one.

url='http://www.google.com/blahblah'

try:
    r = requests.get(url,timeout=3)
    r.raise_for_status()
except requests.exceptions.HTTPError as errh:
    print ("Http Error:",errh)
except requests.exceptions.ConnectionError as errc:
    print ("Error Connecting:",errc)
except requests.exceptions.Timeout as errt:
    print ("Timeout Error:",errt)
except requests.exceptions.RequestException as err:
    print ("OOps: Something Else",err)

Http Error: 404 Client Error: Not Found for url: http://www.google.com/blahblah

vs

url='http://www.google.com/blahblah'

try:
    r = requests.get(url,timeout=3)
    r.raise_for_status()
except requests.exceptions.RequestException as err:
    print ("OOps: Something Else",err)
except requests.exceptions.HTTPError as errh:
    print ("Http Error:",errh)
except requests.exceptions.ConnectionError as errc:
    print ("Error Connecting:",errc)
except requests.exceptions.Timeout as errt:
    print ("Timeout Error:",errt)     

OOps: Something Else 404 Client Error: Not Found for url: http://www.google.com/blahblah

Solution 3

Exception object also contains original response e.response, that could be useful if need to see error body in response from the server. For example:

try:
    r = requests.post('somerestapi.com/post-here', data={'birthday': '9/9/3999'})
    r.raise_for_status()
except requests.exceptions.HTTPError as e:
    print (e.response.text)

Solution 4

Here's a generic way to do things which at least means that you don't have to surround each and every requests call with try ... except:

def requests_call(method, *args, **kwargs):
    requests_method = getattr(requests, method)
    response = None
    exception = None
    try:
        response = requests_method(*args, **kwargs)
    except BaseException as e:
        exception = e
        logger.exception(f'method {method} args {args} kwargs {kwargs}')
    return (response, exception) 

NB

1. Be aware of ConnectionError which is a builtin, nothing to do with the class requests.ConnectionError*. I assume the latter is more common in this context but have no real idea...
2. When examining a non-None returned exception, requests.RequestException, the superclass of all the requests exceptions (including requests.ConnectionError), is not "requests.exceptions.RequestException" according to the docs. Maybe it has changed since the accepted answer.**
3. Obviously this assumes a logger has been configured. Calling logger.exception in the except block will never do any harm and will cause a stack trace to be logged by default (with details of the call which caused the problem)

*I looked at the source code: requests.ConnectionError subclasses the single class requests.RequestException, which subclasses the single class IOError (builtin)

**However at the bottom of this page you find "requests.exceptions.RequestException" at the time of writing (2022-02)... but it links to the above page: confusing.

---

Usage is very simple:

search_response, exception = utilities.requests_call('get',
    f'http://localhost:9200/my_index/_search?q={search_string}')

First you check the response: if it's None something funny has happened and you will have an exception which has to be acted on in some way depending on context (and on the exception). In Gui applications (PyQt5) I usually implement a "visual log" to give some output to the user (and also log simultaneously to the log file), but messages added there should be non-technical. So something like this might typically follow:

if search_response == None:
    # you might check here for (e.g.) a requests.Timeout, tailoring the message
    # accordingly, as the kind of error anyone might be expected to understand
    msg = f'Problem searching on |{search_string}|: no response returned. See log'
    MainWindow.the().visual_log(msg, log_level=logging.ERROR)
    return
if search_response.status_code != 200:
    msg = f'Bad response searching on |{search_string}|. See log'
    MainWindow.the().visual_log(msg, log_level=logging.ERROR)
    log_msg = f'search on |{search_string}| bad response, status code: {search_response.status_code}'
    logger.error(log_msg)
    return
response_json = search_response.json()
...

Given that the stack trace is logged automatically you often need no more than that...

View more solutions