PYTHONPATH environment variable...how do I make every subdirectory afterwards?

Question

I currently do this:

PYTHONPATH=/home/$USER:/home/$USER/respository:/home/$USER/repository/python-stuff

How can I make it so that the PYTHONPATH can include everything subdirectory?

PYTHONPATH = /home/$USER/....and-all-subdirectories

Solution 1

That's not how PYTHONPATH works; PYTHONPATH treats its search path differently from shell PATH. Let's say I do this:

$ mkdir /home/jsmith/python
$ cd /home/jsmith/python
$ touch a.py b.py

This will work, in Python (sys.path will include the current directory):

$ cd /
$ PYTHONPATH=/home/jsmith/python python2.6

Python 2.6.1 (r261:67515, Jul  7 2009, 23:51:51)
>>> import a, b   # Works
>>> quit()

However, subdirectories are treated as packages when __init__.py is present in the directory, and are ignored by PYTHONPATH otherwise:

$ mkdir /home/jsmith/python/pkg
$ cd /home/jsmith/python/pkg
$ touch __init__.py c.py d.py
$ cd /
$ PYTHONPATH=/home/jsmith/python python2.6

Python 2.6.1 (r261:67515, Jul  7 2009, 23:51:51)
>>> import a, b   # Works
>>> import c
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named c

To get at something in that subdirectory, this would work:

>>> from pkg import c   # Works
>>> import pkg.c        # Works

To roll a solution where every subdirectory in your PYTHONPATH is added, you need to explicitly add every folder to PYTHONPATH or sys.path programmatically. This behavior is intentional, and behaves nothing like shell PATH. Given the interpreter's support for packages in this regard, surely there's a better way to accomplish what you're after?

• Understanding imports and PYTHONPATH

Solution 2

It's possible to add subdirectories of a directory to your PYTHONPATH variable using the shell, of course. I currently use something similar to the following in my .bashrc:

export PYTHONPATH="$(find $HOME/ -maxdepth 2 -type d | sed '/\/\./d' | tr '\n' ':' | sed 's/:$//')"

This would include all subdirectories of your user folder to a depth of 2 in the tree. The find command locates the directories ('-type d'), and the following sed and tr commands format the output in the usual way of PATH variables.

Leaving off '-maxdepth 2' would include all subdirectories of your home folder, which is probably quite a lot to search. Perhaps this should only be done in your $HOME/repository/python-stuff directory.

Solution 3

That's not how environment PATH variables work - you give it the top-level directory and it's up to the application to recurse the directory tree if it needs to.

Comments

[Alex]

So, if I have a python file under /home/$USER/myfile.py Can I import this?

[EEAA]

Sure, why couldn't you?

[Jed Smith]

Man, I wish each site would check your other accounts before denying you posting privileges. It sucks starting over with rep, especially with things like the one-URL limit...(I had more references for you, OP)

[mattyb]

I thought it was no longer necessary to add empty init.py files with namespace packages now? For python3 that is.