Namespace vs regular package

Question

What's the difference between a namespace Python package (no __init__.py) and a regular Python package (has an __init__.py), especially when __init__.py is empty for a regular package?

I am curious because recently I've been forgetting to make __init__.py in packages I make, and I never noticed any problems. In fact, they seem to behave identically to regular packages.

Edit: Namespace packages only supported from Python 3.3 (see PEP 420), so naturally, this question only applies to Python 3.

Solution 1

Namespace packages

As of Python 3.3, we get namespace packages. These are a special kind of package that allows you to unify two packages with the same name at different points on your Python-path. For example, consider path1 and path2 as separate entries on your Python-path:

path1
+--namespace
   +--module1.py
   +--module2.py
path2
+--namespace
   +--module3.py
   +--module4.py

with this arrangement you should be able to do the following:

from namespace import module1, module3

thus you get the unification of two packages with the same name in a single namespace.

If either one of them gain an __init__.py that becomes the package - and you no longer get the unification as the other directory is ignored.

If both of them have an __init__.py, the first one in the PYTHONPATH (sys.path) is the one used.

__init__.py used to be required to make directory a package

Namespace packages are packages without the __init__.py.

For an example of a simple package, if you have a directory:

root
+--package
   +--file1.py
   +--file2.py
   ...

While you could run these files independently in the package directory, e.g. with python2 file1.py, under Python 2 you wouldn't be able to import the files as modules in the root directory, e.g.

import package.file1

would fail, and in order for it to work, you at least need this:

package
  +--__init__.py
  +--file1.py
  +--file2.py
  ...

__init__.py initializes the package so you can have code in the __init__.py that is run when the module is first imported:

run_initial_import_setup()

provide an __all__ list of names to be imported,

__all__ = ['star_import', 'only', 'these', 'names']

if the package is imported with the following:

from module import *

or you can leave the __init__.py completely empty if you only want to be able to import the remaining .py files in the directory.

Namespaces with __init__.py using pkgutil:

You could originally use pkgutil, available since Python 2.3. to accomplish adding namespaces, by adding the following into each separate package's __init__.py:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

Setuptools uses a similar method, again, all __init__.py files should contain the following (with no other code):

import pkg_resources
pkg_resources.declare_namespace(__name__)

Namespaces were more thoroughly addressed in PEP 420

See also more discussion on setuptools and Namespaces here:

http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages

Solution 2

Reading link from Aaron, and PEP420, it appears that the fundamental difference between a namespace package and a regular package, beside the obvious difference that a regular package may contain various initialization code in __init__.py, is that a namespace package is a virtual package whose contents can be distributed in various places along Python's lookup path.

For example, given

a/foo/bar.py
b/foo/baz.py

If both b and a are in Python's path, you can import foo.bar and foo.baz freely.

Of course, this begs the question that, if __init__.py is not needed, then all other things being equal, is it better to make a regular package or a namespace package, but is a little off-topic.

Comments

[danijar]

packaging.python.org/guides/packaging-namespace-packages

[variable]

You mention that "While you can run these files independently in the package directory, e.g. with python file1.py, or python3 file1.py, you won't be able to import the files as modules in the root directory" - I am able to import a file as module in root directory - I am using python 3.6 - can you clarify if your answer is specific to an older version of python please?

[Russia Must Remove Putin]

@variable thanks for bringing this one to my attention, please review it and let me know if you think anything is missing?

[Chachni]

I am a bit confused - you have the tree structure: root +--package +--file1.py +--file2.py, and then you do "import module.file1" and say it doesn't work. Did you mean "import package.file1" wouldn't work? Where does the name "module" come from at this point?

[Russia Must Remove Putin]

@Chachni thanks for pointing that out, I have updated the post.