How can one mock/stub python module like urllib

Question

I need to test a function that needs to query a page on an external server using urllib.urlopen (it also uses urllib.urlencode). The server could be down, the page could change; I can't rely on it for a test.

What is the best way to control what urllib.urlopen returns?

Solution 1

Another simple approach is to have your test override urllib's urlopen() function. For example, if your module has

import urllib

def some_function_that_uses_urllib():
    ...
    urllib.urlopen()
    ...

You could define your test like this:

import mymodule

def dummy_urlopen(url):
    ...

mymodule.urllib.urlopen = dummy_urlopen

Then, when your tests invoke functions in mymodule, dummy_urlopen() will be called instead of the real urlopen(). Dynamic languages like Python make it super easy to stub out methods and classes for testing.

See my blog posts at http://softwarecorner.wordpress.com/ for more information about stubbing out dependencies for tests.

Solution 2

I am using Mock's patch decorator:

from mock import patch

[...]

@patch('urllib.urlopen')
def test_foo(self, urlopen_mock):
    urlopen_mock.return_value = MyUrlOpenMock()

Solution 3

Did you give Mox a look? It should do everything you need. Here is a simple interactive session illustrating the solution you need:

>>> import urllib
>>> # check that it works
>>> urllib.urlopen('http://www.google.com/')
<addinfourl at 3082723820L ...>
>>> # check what happens when it doesn't
>>> urllib.urlopen('http://hopefully.doesnotexist.com/')
#-- snip --
IOError: [Errno socket error] (-2, 'Name or service not known')

>>> # OK, let's mock it up
>>> import mox
>>> m = mox.Mox()
>>> m.StubOutWithMock(urllib, 'urlopen')
>>> # We can be verbose if we want to :)
>>> urllib.urlopen(mox.IgnoreArg()).AndRaise(
...   IOError('socket error', (-2, 'Name or service not known')))

>>> # Let's check if it works
>>> m.ReplayAll()
>>> urllib.urlopen('http://www.google.com/')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.5/site-packages/mox.py", line 568, in __call__
    raise expected_method._exception
IOError: [Errno socket error] (-2, 'Name or service not known')

>>> # yay! now unset everything
>>> m.UnsetStubs()
>>> m.VerifyAll()
>>> # and check that it still works
>>> urllib.urlopen('http://www.google.com/')
<addinfourl at 3076773548L ...>

Solution 4

HTTPretty works in the exact same way that FakeWeb does. HTTPretty works in the socket layer, so it should work intercepting any python http client libraries. It's battle tested against urllib2, httplib2 and requests

import urllib2
from httpretty import HTTPretty, httprettified


@httprettified
def test_one():
    HTTPretty.register_uri(HTTPretty.GET, "http://yipit.com/",
                           body="Find the best daily deals")

    fd = urllib2.urlopen('http://yipit.com')
    got = fd.read()
    fd.close()

    assert got == "Find the best daily deals"

Solution 5

In case you don't want to even load the module:

import sys,types
class MockCallable():
  """ Mocks a function, can be enquired on how many calls it received """
  def __init__(self, result):
    self.result  = result
    self._calls  = []

  def __call__(self, *arguments):
    """Mock callable"""
    self._calls.append(arguments)
    return self.result

  def called(self):
    """docstring for called"""
    return self._calls

class StubModule(types.ModuleType, object):
  """ Uses a stub instead of loading libraries """

  def __init__(self, moduleName):
    self.__name__ = moduleName
    sys.modules[moduleName] = self

  def __repr__(self):
    name  = self.__name__
    mocks = ', '.join(set(dir(self)) - set(['__name__']))
    return "<StubModule: %(name)s; mocks: %(mocks)s>" % locals()

class StubObject(object):
  pass

And then:

>>> urllib = StubModule("urllib")
>>> import urllib # won't actually load urllib

>>> urls.urlopen = MockCallable(StubObject())

>>> example = urllib.urlopen('http://example.com')
>>> example.read = MockCallable('foo')

>>> print(example.read())
'foo'

View more solutions

Comments

[user1066101]

Not sure that I like having the fixture under test aware of configuration details... However, this does work.

[user1066101]

Monkeypatches for testing are a handy thing. Indeed, this is probably the canonical "good monkeypatch" example.

[Mu Mind]

visionandexecution.org seems to be down. Is there another link, or is this gone now?

[user220465]

I haven't posted to the blog in a really long time, but I did port it to softwarecorner.wordpress.com

[Tommaso Barbugli]

too bad it does not work when patching module functions :/ (at least not 0.7.2)

[Tommaso Barbugli]

not 100% true, if you import the function before patching it works, otherwise the patching fails silently (no errors, just nothing gets patched :/)

[fatuhoku]

Good point there; patching should throw errors when it's failed to find the relevant module rather than just failing silently.

[fatuhoku]

In 2013, this is definitively the best answer. Let's vote Falcão's awesome library up, guys!

[fatuhoku]

Coming from a Obj-C angle, I was looking for something like OHHTTPStubs for Python. I'm delighted to find HTTPretty.

[Keshi]

Beware! This would mock it out for all instances of urlopen in your test module and other classes in your module if you do not explicitly reset the mocked object back to original value. Ofcourse in this case I am not sure why anyone would want to make network calls in unit tests. I would recommend using something like 'with patch ...' or @patch() which gives you more explicit control on what you are mocking and upto what limits.

[Pratik Khadloya]

It gives me the an error. fixture 'urlopen_mock' not found

[Erik Aronesty]

Close, but the import function won't actually import stuff. So a caller using from urllib import * ... won't get the functions they need

[Florian]

quoting pypi.org/project/mox: "New uses of this library are discouraged. People are encouraged to use pypi.python.org/pypi/mock instead which matches the unittest.mock library available in Python 3."

[Triguna]

Wow., that's the best way to mock anything. Doing the same in JavaScript, good to see the same in Python. Thanks.

[lfagundes]

If you patch urllib.urlopen directly, any references to it that have already been imported by a module will remain unpatched. To avoid that, patch the imported reference instead. ex: patch('mymodule.urlopen')