Import arbitrary python source file. (Python 3.3+)
Solution 1
Found a solution from importlib
test code.
Using importlib.machinery.SourceFileLoader:
>>> import importlib.machinery
>>> loader = importlib.machinery.SourceFileLoader('a_b', '/tmp/a-b.txt')
>>> mod = loader.load_module()
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>
NOTE: only works in Python 3.3+.
UPDATE Loader.load_module
is deprecated since Python 3.4. Use Loader.exec_module
instead:
>>> import types
>>> import importlib.machinery
>>> loader = importlib.machinery.SourceFileLoader('a_b', '/tmp/a-b.txt')
>>> mod = types.ModuleType(loader.name)
>>> loader.exec_module(mod)
>>> mod
<module 'a_b'>
>>> import importlib.machinery
>>> import importlib.util
>>> loader = importlib.machinery.SourceFileLoader('a_b', '/tmp/a-b.txt')
>>> spec = importlib.util.spec_from_loader(loader.name, loader)
>>> mod = importlib.util.module_from_spec(spec)
>>> loader.exec_module(mod)
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>
Solution 2
Updated for Python >= 3.8:
Short version:
>>> # https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly
>>> import importlib.util, sys
>>> spec = importlib.util.spec_from_file_location(modname, fname)
>>> module = importlib.util.module_from_spec(spec)
>>> sys.modules[modname] = module
>>> spec.loader.exec_module(module)
Full version:
>>> import importlib.util
>>> import sys
>>> from pathlib import Path
>>> from typing import TYPE_CHECKING
>>>
>>>
>>> if TYPE_CHECKING:
... import types
...
...
>>> def import_source_file(fname: str | Path, modname: str) -> "types.ModuleType":
... """
... Import a Python source file and return the loaded module.
... Args:
... fname: The full path to the source file. It may container characters like `.`
... or `-`.
... modname: The name for the loaded module. It may contain `.` and even characters
... that would normally not be allowed (e.g., `-`).
... Return:
... The imported module
... Raises:
... ImportError: If the file cannot be imported (e.g, if it's not a `.py` file or if
... it does not exist).
... Exception: Any exception that is raised while executing the module (e.g.,
... :exc:`SyntaxError). These are errors made by the author of the module!
... """
... # https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly
... spec = importlib.util.spec_from_file_location(modname, fname)
... if spec is None:
... raise ImportError(f"Could not load spec for module '{modname}' at: {fname}")
... module = importlib.util.module_from_spec(spec)
... sys.modules[modname] = module
... try:
... spec.loader.exec_module(module)
... except FileNotFoundError as e:
... raise ImportError(f"{e.strerror}: {fname}") from e
... return module
...
>>> import_source_file(Path("/tmp/my_mod.py"), "my_mod")
<module 'my_mod' from '/tmp/my_mod.py'>
Original answer for Python 3.5 and 3.6
Shorter version of @falsetru 's solution:
>>> import importlib.util
>>> spec = importlib.util.spec_from_file_location('a_b', '/tmp/a-b.py')
>>> mod = importlib.util.module_from_spec(spec)
>>> spec.loader.exec_module(mod)
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>
I tested it with Python 3.5 and 3.6.
According to the comments, it does not work with arbitrary file extensions.
Solution 3
Similar to @falsetru but for Python 3.5+ and accounting for what the importlib
doc states on using importlib.util.module_from_spec
over types.ModuleType
:
This function [
importlib.util.module_from_spec
] is preferred over usingtypes.ModuleType
to create a new module as spec is used to set as many import-controlled attributes on the module as possible.
We are able to import any file with importlib
alone by modifying the importlib.machinery.SOURCE_SUFFIXES
list.
import importlib
importlib.machinery.SOURCE_SUFFIXES.append('') # empty string to allow any file
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
# if desired: importlib.machinery.SOURCE_SUFFIXES.pop()
Solution 4
importlib
helper function
Here is a convenient, ready-to-use helper to replace imp
, with an example. The technique is the same as that of https://stackoverflow.com/a/19011259/895245 , this is just providing a more convenient function.
main.py
#!/usr/bin/env python3
import os
import importlib
def import_path(path):
module_name = os.path.basename(path).replace('-', '_')
spec = importlib.util.spec_from_loader(
module_name,
importlib.machinery.SourceFileLoader(module_name, path)
)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
sys.modules[module_name] = module
return module
notmain = import_path('not-main')
print(notmain)
print(notmain.x)
not-main
x = 1
Run:
python3 main.py
Output:
<module 'not_main' from 'not-main'>
1
I replace -
with _
because my importable Python executables without extension have hyphens as in my-cmd
. This is not mandatory, but produces better module names like my_cmd
.
This pattern is also mentioned in the docs at: https://docs.python.org/3.7/library/importlib.html#importing-a-source-file-directly
I ended up moving to it because after updating to Python 3.7, import imp
prints:
DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
and I don't know how to turn that off, this was asked at:
Tested in Python 3.7.3.
Solution 5
after many failure solutions this one works for me
def _import(func,*args):
import os
from importlib import util
module_name = "my_module"
BASE_DIR = "wanted module directory path"
path = os.path.join(BASE_DIR,module_name)
spec = util.spec_from_file_location(func, path)
mod = util.module_from_spec(spec)
spec.loader.exec_module(mod)
return getattr(mod,func)(*args)
and to call it just write the function name and it's parameters _import("function",*args)
Related videos on Youtube
Comments
-
falsetru about 2 years
How can I import an arbitrary python source file (whose filename could contain any characters, and does not always ends with
.py
) in Python 3.3+?I used
imp.load_module
as follows:>>> import imp >>> path = '/tmp/a-b.txt' >>> with open(path, 'U') as f: ... mod = imp.load_module('a_b', f, path, ('.py', 'U', imp.PY_SOURCE)) ... >>> mod <module 'a_b' from '/tmp/a-b.txt'>
It still works in Python 3.3, but according to
imp.load_module
documentation, it is deprecated:Deprecated since version 3.3: Unneeded as loaders should be used to load modules and find_module() is deprecated.
and
imp
module documentation recommends to useimportlib
:Note New programs should use importlib rather than this module.
What is the proper way to load an arbitrary python source file in Python 3.3+ without using the deprecated
imp.load_module
function?-
Brett Cannon about 10 yearsCan I ask why you are doing this? I'm the maintainer of importlib and I have been trying to get answers from folks as to why they use
imp.load_module()
over a straight import statement. Do you expect to import the module by name later (e.g.import a_b
)? Do you care that any custom importers won't be used in this approach? Do you expect the module to be full-featured (e.g. define__name__
and__loader__
)? -
falsetru about 10 years@BrettCannon, A third-party program regularly (once a hour) modify a text file that contains python statements (mainly
THIS='blah'
like lines). The name of the file is not ended with.py
. My program read that file. -
falsetru about 10 years@BrettCannon, I'm not aware of custom importers. I don't care the module to be full-featured.
-
Brett Cannon about 10 yearsIOW using Python as a really simple data structure format. Thanks for the info!
-
falsetru almost 10 years@downvoter, Could you explain why?
-
Andrew Miner over 6 years@BrettCannon — I just ran into a case where I needed to import some Python code from within a directory which was named as a version number (e.g., "v1.0.2"). While possible, it would be highly undesirable to rename the directory. I wound up using stefan-scherfke's solution below.
-
Ciro Santilli OurBigBook.com about 5 years
-
Brad over 4 years@BrettCannon: Enaml is a use-case where one needs to use a different extension (in this case,
*.enaml
). Enaml is a superset of Python that allows for declarative markup (useful for creating responsive GUIs). Enaml runs with Python and has it's own import hooks (e.g.,enaml.import_hooks
) to allow for loading Enaml files in Python programs. Importing Enaml files typically requires doing so within an Enaml context manager (e.g.,with enaml.imports(): import ...
). However, sometimes we want to useimportlib
to load from an arbitrary Enaml source file. -
Ciro Santilli OurBigBook.com over 3 yearsDoes this answer your question? Import a python module without the .py extension
-
falsetru over 3 years@CiroSantilli郝海东冠状病六四事件法轮功, Answers there mostly focused on python 2.x solution (imp), but I wanted solution that works Python 3.3+. Answers there that solve my question come after my own answer. (2013 vs 2016,2017,2019) This one answered my question.
-
Ciro Santilli OurBigBook.com over 3 yearsIt's true, retracted.
-
-
falsetru over 10 yearsDownvoter: How can I improve the answer? If you have a better way to accomplish what I want, please let me know.
-
Eryk Sun over 10 yearsThere's a helpful warning that
load_module
ignores viawarnings.catch_warnings
. If you instead usemod = imp.load_source('a_b', '/tmp/a-b.txt')
, it raises the following warning (use-Wall
):DeprecationWarning: imp.load_source() is deprecated; use importlib.machinery.SourceFileLoader(name, pathname).load_module() instead
. -
falsetru over 10 years@eryksun, You're right. Thank you for the comment. BTW, Python 3.4(rc1) does not display the alternative usage unlike Python 3.3.x.
-
falsetru over 7 years
importlib.util.spec_from_file_location(..)
returnsNone
for me; causing an exception for the followingimportlib.util.module_from_spec(..)
call. (See i.imgur.com/ZjyFhif.png) -
falsetru over 7 years
importlib.util.spec_from_file_location
works for known file name extensions (.py
,.so
, ..), but not for others (.txt
...) -
Stefan Scherfke over 7 yearsOh, I’m using it only with Python files but modified my example to look like the one above and did not test it … I updated it.
-
Matthew D. Scholefield about 6 yearsWhat's the difference between the first and the second example at the bottom?
-
falsetru about 6 years@MatthewD.Scholefield ways to get module objects are different. Using Module type directly or using utility.
-
mxxk almost 6 yearsInterestingly enough, while this hack of appending the empty string to the list of source suffixes works great for importing renamed Python source modules, the equivalent for importing renamed extension modules does not work... That is, using
importlib.machinery.EXTENSION_SUFFIXES.append('')
still makesimportlib.util.spec_from_file_location
return None. -
mxxk almost 6 years@falsetru - great work. Would be good to add that
SourceFileLoader
only understands pure Python source modules. For loading extension modules (.so / .dll files) with an arbitrary suffix, the sibling classExtensionFileLoader
is the way to go :) -
Alex Walczak over 5 yearspresumably,
importlib.util.spec_from_file_location
should still work with extensions if you specify a loader -
e-info128 over 5 yearsHowto pass arguments in module constructor?
-
falsetru over 5 years@e-info128 What is module constructor?
-
ihavenoidea over 3 years@falsetru I know this deviates a little from OP's question but, what if I have a (for example) class implementation that is held inside a string instead of in a file. Ex:
my_class = """class Test: def __init__(self): self.x = 5 def print_number(self): print(self.x)"""
Is there a way to programmatically import this class inside the string to the namespace? -
falsetru over 3 years@ihavenoidea, Please post a separated question, so that others can answer you, and other users can also read answers.
-
ihavenoidea over 3 years@falsetru In fact I have here, no answer so far. I posted here in the comments because I bumped into your answer after posting my question. If you know how to do this, I'd appreciate it!
-
SurpriseDog about 3 yearsI can't get this to load the source code from a class. Can you help me with my question here: stackoverflow.com/q/67663614/11343425 ? Thanks
-
Jon about 3 years@mxxk that's because, if you look at the source, it's only checking if the suffix is present for files named
__init__
-
YoomarXD almost 3 yearsHow do I pass all the constants, methods and classes from mod to the main scope? I tried
from mod import *
but obviously didn't work. -
falsetru almost 3 years@YoomarXD,
mod['name_you_want_access']
-
YoomarXD almost 3 years@falsetru I meant to pass mod.constant1, mod.method2, etc. to the global scope simulating the behavior of
from foo import *
. But nvm, already managed to archive it by:for key, val in [p for p in mod.__dict__.items() if not p[0].startswith('__')]: globals()[key] = val
-
falsetru almost 3 years@YoomarXD,
globals().update({key: value for key, value in vars(mod).items() if not key.startswith('__')})
, BTW, I don't recommend to pollute module namespace at runtime.