Cleanly setting max_retries on Python requests get or post method

Question

Related to older version of requests question: Can I set max_retries for requests.request?

I have not seen an example to cleanly incorporate max_retries in a requests.get() or requests.post() call.

Would love a

requests.get(url, max_retries=num_max_retries))

implementation

Solution

A quick search of the python-requests docs will reveal exactly how to set max_retries when using a Session.

To pull the code directly from the documentation:

import requests
s = requests.Session()
a = requests.adapters.HTTPAdapter(max_retries=3)
b = requests.adapters.HTTPAdapter(max_retries=3)
s.mount('http://', a)
s.mount('https://', b)
s.get(url)

---

What you're looking for, however, is not configurable for several reasons:

1. Requests no longer provides a means for configuration

2. The number of retries is specific to the adapter being used, not to the session or the particular request.

3. If one request needs one particular maximum number of requests, that should be sufficient for a different request.

This change was introduced in requests 1.0 over a year ago. We kept it for 2.0 purposefully because it makes the most sense. We also will not be introducing a parameter to configure the maximum number of retries or anything else, in case you were thinking of asking.

---

Edit Using a similar method you can achieve a much finer control over how retries work. You can read this to get a good feel for it. In short, you'll need to import the Retry class from urllib3 (see below) and tell it how to behave. We pass that on to urllib3 and you will have a better set of options to deal with retries.

from requests.packages.urllib3 import Retry
import requests

# Create a session
s = requests.Session()

# Define your retries for http and https urls
http_retries = Retry(...)
https_retries = Retry(...)

# Create adapters with the retry logic for each
http = requests.adapters.HTTPAdapter(max_retries=http_retries)
https = requests.adapters.HTTPAdapter(max_retries=https_retries)

# Replace the session's original adapters
s.mount('http://', http)
s.mount('https://', https)

# Start using the session
s.get(url)

Comments

[Jari Turkia]

This is the best suggestion I found about retries: peterbe.com/plog/best-practice-with-retries-with-requests