Unit testing a python app that uses the requests library

Question

I am writing an application that performs REST operations using Kenneth Reitz's requests library and I'm struggling to find a nice way to unit test these applications, because requests provides its methods via module-level methods.

What I want is the ability to synthesize the conversation between the two sides; provide a series of request assertions and responses.

Solution 1

It is in fact a little strange that the library has a blank page about end-user unit testing, while targeting user-friendliness and ease of use. There's however an easy-to-use library by Dropbox, unsurprisingly called responses. Here is its intro post. It says they've failed to employ httpretty, while stating no reason of the fail, and written a library with similar API.

import unittest

import requests
import responses


class TestCase(unittest.TestCase):

  @responses.activate  
  def testExample(self):
    responses.add(**{
      'method'         : responses.GET,
      'url'            : 'http://example.com/api/123',
      'body'           : '{"error": "reason"}',
      'status'         : 404,
      'content_type'   : 'application/json',
      'adding_headers' : {'X-Foo': 'Bar'}
    })

    response = requests.get('http://example.com/api/123')

    self.assertEqual({'error': 'reason'}, response.json())
    self.assertEqual(404, response.status_code)

Solution 2

If you use specifically requests try httmock. It's wonderfully simple and elegant:

from httmock import urlmatch, HTTMock
import requests

# define matcher:
@urlmatch(netloc=r'(.*\.)?google\.com$')
def google_mock(url, request):
    return 'Feeling lucky, punk?'

# open context to patch
with HTTMock(google_mock):
    # call requests
    r = requests.get('http://google.com/')
print r.content  # 'Feeling lucky, punk?'

If you want something more generic (e.g. to mock any library making http calls) go for httpretty.

Almost as elegant:

import requests
import httpretty

@httpretty.activate
def test_one():
    # define your patch:
    httpretty.register_uri(httpretty.GET, "http://yipit.com/",
                        body="Find the best daily deals")
    # use!
    response = requests.get('http://yipit.com')
    assert response.text == "Find the best daily deals"

HTTPretty is far more feature-rich - it offers also mocking status code, streaming responses, rotating responses, dynamic responses (with a callback).

Solution 3

You could use a mocking library such as Mocker to intercept the calls to the requests library and return specified results.

As a very simple example, consider this class which uses the requests library:

class MyReq(object):
    def doSomething(self):
        r = requests.get('https://api.github.com', auth=('user', 'pass'))
        return r.headers['content-type']

Here's a unit test that intercepts the call to requests.get and returns a specified result for testing:

import unittest
import requests
import myreq

from mocker import Mocker, MockerTestCase

class MyReqTests(MockerTestCase):
    def testSomething(self):
        # Create a mock result for the requests.get call
        result = self.mocker.mock()
        result.headers
        self.mocker.result({'content-type': 'mytest/pass'})

        # Use mocker to intercept the call to requests.get
        myget = self.mocker.replace("requests.get")
        myget('https://api.github.com', auth=('user', 'pass'))
        self.mocker.result(result)

        self.mocker.replay()

        # Now execute my code
        r = myreq.MyReq()
        v = r.doSomething()

        # and verify the results
        self.assertEqual(v, 'mytest/pass')
        self.mocker.verify()

if __name__ == '__main__':
    unittest.main()

When I run this unit test I get the following result:

.
----------------------------------------------------------------------
Ran 1 test in 0.004s

OK

Solution 4

Missing from these answers is requests-mock.

From their page:

>>> import requests
>>> import requests_mock

As a context manager:

>>> with requests_mock.mock() as m:

...     m.get('http://test.com', text='data')
...     requests.get('http://test.com').text
...
'data'

Or as a decorator:

>>> @requests_mock.mock()
... def test_func(m):
...     m.get('http://test.com', text='data')
...     return requests.get('http://test.com').text
...
>>> test_func()
'data'

Solution 5

using mocker like in srgerg's answer:

def replacer(method, endpoint, json_string):
    from mocker import Mocker, ANY, CONTAINS
    mocker = Mocker()
    result = mocker.mock()
    result.json()
    mocker.count(1, None)
    mocker.result(json_string)
    replacement = mocker.replace("requests." + method)
    replacement(CONTAINS(endpoint), params=ANY)
    self.mocker.result(result)
    self.mocker.replay()

For the requests library, this would intercept the request by method and endpoint you're hitting and replace the .json() on the response with the json_string passed in.

View more solutions

Comments

[kgr]

So you need to mock out the REST server?

[John Mee]

why does that make unittest unsuitable? Check how the library does its own unit tests; might offer ideas.

[Romløk]

FWIW, the Requests library does its own tests using live URLs (github.com, the authors own domain, etc.).

[mccc]

That httpretty thing is straight from out of this world, thanks!

[Thomas Fauskanger]

Do you have any knowledge of how to make this work with pytest? I tried the exact example you're citing. Ref.: stackoverflow.com/questions/47703748/…

[Unapiedra]

If I remember correctly, I used the decorator. And I think this also worked with pytest.

[Thomas Fauskanger]

I made it work with decorators, but it appears (on my system) that it's in conflict with some other argument, so I had to pass a kw argument to the Mocker, as mentioned in docs. Not sure if this has to do with pytest, but the error that came up mentioned fixtures. Thanks for getting back to the issue.

[Mohamed Ragab]

Updated URL for the responses intro post

[bfontaine]

Interestingly, since this was posted as an answer, David Cramer –who authored this library– moved on and founded Sentry, and moved the library with him. This is why on GitHub it’s under the getsentry org.