reloading module which has been imported to another module

Question

Let's face it, the whole business of reloading python code after changing it is a mess. I figured out awhile back that calling import <module> at the interpreter is better than from <module> import <class/function>, because then I can call reload(module) to get updated code.

But I have more complicated issues now. So I have this file, module1.py, and at the top it says:

from module2 import <class1>, <function1>, etc.

And then I go and change code inside module2. Turns out that calling reload(module1) will not reload the code changed in module2, even though code from module2 is imported at the top of module1. Is there any way to reload everything without restarting the interpreter?

Before anyone gets on my case about style, I'll just say that:

1. I only call reload from the interpreter, never in active code. This question concerns when I'm testing new code.
2. I never call from <module> import *, I know that destroys readability

Solution 1

Have a look into IPython. It has the autoreload extension that automatically reloads modules during the interpreter session before calling functions within. I cite the example from the landing page:

In [1]: %load_ext autoreload

In [2]: %autoreload 2

In [3]: from foo import some_function

In [4]: some_function()
Out[4]: 42

In [5]: # open foo.py in an editor and change some_function to return 43

In [6]: some_function()
Out[6]: 43

Solution 2

To reload a module, you have to use reload, and you have to use it on the module you want to reload. Reloading a module doesn't recursively reload all modules imported by that module. It just reloads that one module.

When a module is imported, a reference to it is stored, and later imports of that module re-use the already-imported, already-stored version. When you reload module1, it re-runs the from module2 import ... statement, but that just reuses the already-imported version of module2 without reloading it.

The only way to fix this is to change your code so it does import module2 instead of (or in addition to) from module2 import .... You cannot reload a module unless the module itself has been imported and bound to a name (i.e., with an import module statement, not just a from module import stuff statement).

Note that you can use both forms of the import, and reloading the imported module will affect subsequent from imports. That is, you can do this:

>>> import module
>>> from module import x
>>> x
2
# Change module code here, changing x to 3
>>> reload(module)
>>> from module import x
>>> x
3

This can be handy for interactive work, since it lets you use short, unprefixed names to refer to what you need, while still being able to reload the module.

Solution 3

Rather than getting better at reloading modules, you could get better at restarting the interpreter. For example, you can put your setup code into its own file, and then run it like this:

$ python -i setup.py
>>>

This will run setup.py, then leave you at the interactive prompt. Or, rather than doing a lot of work in the interactive prompt, write automated tests that do your work for you.

You are right, reloading modules in Python is a mess. The semantics of the language make it difficult to change code while the process is running. Learn not to need reloading modules, you'll be happier.

Solution 4

Ok, I'm not sure that qualifies as an answer without a change to the code, but... at least, that doesn't involve a change to module1.

You can use some module wrapper class, that saves loaded modules before and after loading module1 and that provides a reload method, something like that:

import sys

class Reloader(object):
    def __init__(self, modulename):
        before = sys.modules.keys()
        __import__(modulename)
        after = sys.modules.keys()
        names = list(set(after) - set(before))
        self._toreload = [sys.modules[name] for name in names]

    def do_reload(self):
        for i in self._toreload:
            reload(i)

Then load module1 with:

reloader = Reloader('module1')

Atfer that you can modify module2 and reload it in interpreter with:

reloader.do_reload()

Solution 5

Don't forget that an import is really just assigning a name in a namespace. So, you could reassign that name after reloading:

>>> reload(module2)
>>> module1.class1 = module2.class1

Now the class1 object inside module1 refers to the reloaded version from module2.

View more solutions

Comments

[Eric Wilson]

Interesting idea, but sometimes the command history of the interpreter is useful. Do you really destroy it every time you change your code?

[utapyngo]

How about IPython? It saves the command history between sessions.

[Eric Wilson]

Thanks, but I'm not going to be using IPython or Jython.

[Admin]

It would make more sense to use the PYTHONSTARTUP environment variable. @EricWilson, you can just read that same startup script with readline.read_history_file("setup.py") to add all the commands to your history. @utapyngo, you don't need IPython for that; the readline module provides that capability: atexit.register(readline.write_history_file, hist_file), where hist_file is the name of the file that you want to contain your history.

[BrenBarn]

That will not solve the proble the OP is asking about. That function works by looking for all global variables in the module that refer to other modules. These will be created by statements like import someModule. But if you do from someModule import something, no variable referring to the module itself is created, so that rreload function will not reload someModule.

[Johannes P]

For completeness, here is the reference for reload: docs.python.org/2/library/functions.html#reload

[Eric Wilson]

I'm happy to award the bounty here, because discovering IPython was worth far more than 50 rep. Thanks.

[AlbertFerras]

that is true, I didn't think about it