When to use `raise_for_status` vs `status_code` testing

Question

I have always used:

r = requests.get(url)
if r.status_code == 200:
    # my passing code
else:
    # anything else, if this even exists

Now I was working on another issue and decided to allow for other errors and am instead now using:

try:
    r = requests.get(url)
    r.raise_for_status()
except requests.exceptions.ConnectionError as err:
    # eg, no internet
    raise SystemExit(err)
except requests.exceptions.HTTPError as err:
    # eg, url, server and other errors
    raise SystemExit(err)
# the rest of my code is going here

With the exception that various other errors could be tested for at this level, is one method any better than the other?

Solution 1

Response.raise_for_status() is just a built-in method for checking status codes and does essentially the same thing as your first example.

There is no "better" here, just about personal preference with flow control. My preference is toward try/except blocks for catching errors in any call, as this informs the future programmer that these conditions are some sort of error. If/else doesn't necessarily indicate an error when scanning code.

---

Edit: Here's my quick-and-dirty pattern.

import time

import requests
from requests.exceptions import HTTPError

url = "https://theurl.com"
retries = 3

for n in range(retries):
    try:
        response = requests.get(url)
        response.raise_for_status()

        break

    except HTTPError as exc:
        code = exc.response.status_code
        
        if code in [429, 500, 502, 503, 504]:
            # retry after n seconds
            time.sleep(n)
            continue

        raise
      

However, in most scenarios, I subclass requests.Session, make a custom HTTPAdapter that handles exponential backoffs, and the above lives in an overridden requests.Session.request method. An example of that can be seen here.

Solution 2

Almost always, raise_for_status() is better.

The main reason is that there is a bit more to it than testing status_code == 200, and you should be making best use of tried-and-tested code rather than creating your own implementation.

For instance, did you know that there are actually five different 'success' codes defined by the HTTP standard? Four of those 'success' codes will be misinterpreted as failure by testing for status_code == 200.

Solution 3

If you are not sure, follow the Ian Goldby's answer.

...however please be aware that raise_for_status() is not some magical or exceptionally smart solution - it's a very simple function that decodes the response body and throws an exception for HTTP codes 400-599, distinguishing client-side and server-side errors (see its code here).

And especially the client-side error responses may contain valuable information in the response body that you may want to process. For example a HTTP 400 Bad Request response may contain the error reason.

In such a case it may be cleaner to not use raise_for_status() but do cover all the cases it does by yourself.

Example code

try:
    r = requests.get(url)

    # process the specific codes from the range 400-599
    # that you are interested in first
    if r.status_code == 400:
        invalid_request_reason = r.text
        print(f"Your request has failed because: {invalid_request_reason}")
        return
    # this will handle all other errors
    elif r.status_code > 400:
        print(f"Your request has failed with status code: {r.status_code}")
        return

except requests.exceptions.ConnectionError as err:
    # eg, no internet
    raise SystemExit(err)

# the rest of my code is going here

Real-world use case

PuppetDB's API using the Puppet Query Language (PQL) responds with a HTTP 400 Bad Request to a syntactically invalid query with a very precise info where is the error.

Request query:

nodes[certname] { certname == "bastion" }

Body of the HTTP 400 response:

PQL parse error at line 1, column 29:

nodes[certname] { certname == "bastion" }
                            ^

Expected one of:

[
false
true
#"[0-9]+"
-
'
"
#"\s+"

See my Pull Request to an app that uses this API to make it show this error message to a user here, but note that it doesn't exactly follow the example code above.

Solution 4

Better is somewhat subjective; both can get the job done. That said, as a relatively inexperienced programmer I prefer the Try / Except form. For me, the T / E reminds me that requests don't always give you what you expect (in a way that if / else doesn't - but that could just be me).

raise_for_status() also lets you easily implement as many or as few different actions for the different error types (.HTTPError, .ConnectionError) as you need. In my current project, I've settled on the form below, as I'm taking the same action regardless of cause, but am still interested to know the cause:

    try:
        ...
    except requests.exceptions.RequestException as e:
        raise SystemExit(e) from None

Toy implementation:

import requests

def http_bin_repsonse(status_code):
    sc = status_code
    try:
        url = "http://httpbin.org/status/" + str(sc)
        response = requests.post(url)
        response.raise_for_status()

        p = response.content

    except requests.exceptions.RequestException as e:
        print("placeholder for save file / clean-up")
        raise SystemExit(e) from None
    
    return response, p

response, p = http_bin_repsonse(403)
print(response.status_code)

View more solutions

Comments

[Ekrem Dinçel]

Second one may be more readable because of exception names instead of status codes, but I think first one is simpler and you can write comments about meaning of status codes.

[Madivad]

@EkremDİNÇEL I'm leaning that way because in this instance I'm using a system that doesn't always have an internet connection and needed to add the test in, and quite frankly I preferred the way this looked :)

[Madivad]

I kind of like the try/except blocks, I just find that they get messy when 2 or 3 (or more) become nested. It's bad enough for nested if blocks, but multiple try/except/except/else/finally blocks become quite unreadable. Having said that, I'm growing to like them :)

[Sam Morgan]

I agree. I go above-and-beyond to reduce nesting wherever possible. A mountain of my coding time goes into readability, "Code is read more often than it is written" and all. I've found that if nested or multiple (more than 2) try/except blocks, or really any deep nesting/large conditional blocks exist, this is almost always a trigger for refactoring.

[Greg Dubicki]

To clarify, OP's first example accepts only HTTP 200 as a successful condition while the second de facto accepts all HTTP codes < 400 (raise_for_status() throws an exception on codes 400-599), so it's very roughly "the same thing".

[Greg Dubicki]

(Requests library follows the HTTP 3xx redirects by default (except for HEAD), so 300-399 codes are processed the same way in both cases.)

[faridSam]

Yes, it's always a good idea to look inside the response to get more information about what went wrong and to make this available somewhere for troubleshooting. That same response is also available through the exception raised by raise_for_status() - ex.response.status_code, ex.response.text, etc.