How to mock os.walk in python with a temporary filesystem?

Question

I'm trying to test some code that uses os.walk. I want to create a temporary, in-memory filesystem that I can populate with sample (empty) files and directories that os.walk will then return. This should save me the complexity of mocking os.walk calls to simulate recursion.

Specifically, the code I want to test is:

if recursive:
    log.debug("Recursively searching for files under %s" % path)

    for (dir_path, dirs, files) in os.walk(path):
        log.debug("Found %d files in %s: %s" % (len(files), path, files))
        for f in [os.path.join(dir_path, f) for f in files
                  if not re.search(exclude, f)]:
            yield f
else:
    log.debug("Non-recursively searching for files under %s" % path)

    for (dir_path, dirs, files) in os.walk(path):
        log.debug("Found %d files in %s: %s" % (len(files), path, files))
        for f in [os.path.join(dir_path, f) for f in files
                    if not re.search(exclude, f)]:
            yield f

Is this possible in python?

Solution

No. os.walk() is constructed entirely around os.listdir(), with assistance of os.path.islink() and os.path.isdir(). These are essentially system calls, so you'd have to mock your filesystem at the system level. Unless you want to write a FUSE plugin this is not going to be easy to mock.

All os.walk() needs to return is a list of tuples, really. Unless you are testing manipulating the dirs component, it couldn't be more simple:

with mock.patch('os.walk') as mockwalk:
    mockwalk.return_value = [
        ('/foo', ('bar',), ('baz',)),
        ('/foo/bar', (), ('spam', 'eggs')),
    ]

This would mock the following directory structure:

/foo
 ├── baz
 └── bar 
     ├── spam
     └── eggs