Dynamic Loading of Python Modules

Question

I'm trying to dynamically load modules I've created.

Right now this works properly:

import structures.index

But if I try the same thing by importing it dynamically, it fails.

struct = __import__("structures.index")

Error supplied is:

Error ('No module named structures.index',)

Any ideas why?

---

Edit: When using full scope (it sort of works?):

struct = __import__("neoform.structures.index")

This doesn't throw any errors, however, it isn't loading the index module, it's loading the "neoform" module instead.

The result of "struct" is:

<module 'neoform' from '/neoform/__init__.py'>

Also, as a side question, how can I then instantiate a class within a dynamically loaded module? (assuming all the modules contain a common class name).

Edit: Solution: (thanks coonj & Rick) This ended up being what worked. Not sure why (yet), but the fromlist had to be something "anything apparently, since it worked when I put the letter "a" as a value (strange, given that the file only had 1 class in it).

def get_struct_module(self, name):
    try:
        return = __import__("neoform.structures." + name, fromlist='*')
    except ImportError, e:
        self.out.add("Could not load struct: neoform.structure." + name + "\n\n" + "Error " + str(e.args))

Solution 1

I'm not sure what "it fails" means, so I'll just mention that __import__('structures.index') should, in fact, work, but it doesn't assign the module name in the current scope. To do that (and then use a class in the dynamically imported module), you'll have to use:

structures = __import__('structures.index')
structures.index.SomeClass(...)

The complete details on __import__ are available here.

Edit: (based on question edit)

To import neoform.structures.index, and return the index module, you would do the following:

structures = __import__('neoform.structures.index', 
                        fromlist=['does not in fact matter what goes here!'])

So if you have a list of package names packages, you can import their index modules and instantiate some MyClass class for each using the following code:

modules = [ __import__('neoform.%s.index' % pkg, fromlist=['a']) 
            for pkg in packages ]
objects = [ m.MyClass() for m in modules ]

Solution 2

To import sub-modules, you need to specify them in the fromlist arg of __import__()
Fo example, the equivalent of:

import structures.index

is:

structures = __import__('structures', fromlist=['index'])

To do this in a map is a little more tricky...

import mod1.index
import mod2.index
import mod3.index

For those imports, you would want to define a new function to get the index sub-module from each module:

def getIndexMods(mod_names):
  mod_list = map(lambda x: __import__(x, fromlist='index'))
  index_mods = [mod.index for mod in mod_list]
  return index_mods

Now, you can do this to get references to all index modules:

index_mods = getIndexMods(['mod1', 'mod2', 'mod3'])

Also, if you want to grab sub-modules that are not named 'index' then you could do this:

mod1, mod2, mod3 = map(lambda x,y: __import__(x, fromlist=y), 
  ['mod1', 'mod2', 'mod3'], ['index1', 'index2', 'index3'])

Solution 3

Use full scope ("neoform.structures.index") with this helper method.

def import_module(name):
    mod = __import__(name)
    components = name.split('.')
    for comp in components[1:]:
        mod = getattr(mod, comp)
    return mod

module = import_module("neoform.structures.index")
# do stuff with module

Solution 4

Java programmer here, but I think you need the imp module

Solution 5

>>> import imp
>>> fm = imp.find_module('index', ['./structures']) # for submodule
>>> mymod = imp.load_module('structures.index', *fm)
>>> mymod
<module 'structures.index' from './structures/index.pyc'>
>>> x = mymod.insideIndex()
Initialising index class...

Voila!

View more solutions

Comments

[dfa]

just noticied the implementation of import using the imp module; thanks :)

[Jason Coon]

I'm guessing he wants to import multiple modules dynamically. Ian? Any response to this?

[Jason Coon]

And by "dynammic" I'm assuming Ian means that he doesn't know what he is importing until runtime.

[Ian]

I'm doing it this way because the string is going to be a variable called from a database.. i simply use a string to simplify the example.

[Ian]

That would be annoying. The reason I'm storing it in a database is because I'm making a mod_wsgi site. Each modfule is a separate page on the site. I've already done this sort of thing with PHP, and I've had more than 300 structures on certain sites, each having it's own hierarchy, permissions, and other attributes. Storing it in the db is the best way I've found.

[Rick Copeland]

You actually don't need to use fromlist to import sub-modules; try "__import__('os.path')". Works just fine (though it returns os, not os.path.) fromlist emulates "from os import path" which is a different beast.

[Jason Coon]

How are you calling it? Maybe are you trying to do this?: from structures import index

[Jason Coon]

I updated my answer, and I made some assumptions based o comments you left on other answers.

[user1066101]

@Ian: Python is not PHP. The techniques that work for PHP do not always translate to Python. I think what you are doing won't work out very well for Python. Look at TurboGears or web.py for examples of how do to this in Python.