Why does from scipy import spatial work, while scipy.spatial doesn't work after import scipy?

Question

I would like to use scipy.spatial.distance.cosine in my code. I can import the spatial submodule if I do something like import scipy.spatial or from scipy import spatial, but if I simply import scipy calling scipy.spatial.distance.cosine(...) results in the following error: AttributeError: 'module' object has no attribute 'spatial'.

What is wrong with the second approach?

Solution 1

Importing a package does not import submodule automatically. You need to import submodule explicitly.

For example, import xml does not import the submodule xml.dom

>>> import xml
>>> xml.dom
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'dom'
>>> import xml.dom
>>> xml.dom
<module 'xml.dom' from 'C:\Python27\lib\xml\dom\__init__.pyc'>

There's an exception like os.path. (os module itself import the submodule into its namespace)

>>> import os
>>> os.path
<module 'ntpath' from 'C:\Python27\lib\ntpath.pyc'>

Solution 2

That's because scipy is a package, not a module. When you import a package, you don't actually load the modules inside, and thus package.module causes an error.

However, import package.module would work, because it loads the module, not the package.

This is the standard behavior for most import statements, but there are a few exceptions.

Here is the same case for urllib in Python 3:

>>> import urllib
>>> dir(urllib)
['__builtins__', '__cached__', '__doc__', '__file__', '__initializing__', '__loader__', '__name__', '__package__', '__path__', 'error', 'parse', 'request', 'response']

See? there is no submodules there. To access its submodule, we ask for the submodule:

>>> import urllib.request
>>> 

Hope this simple explanation helps!

Comments

[Robert Kern]

@alKid and @falsetru gave good answers, and you should accept one of them. To explain why this is the case for scipy in particular, the reason that we do not import all subpackages is that there are a lot of them and many have large extension modules that consume quite a bit of time to load. Most programs don't need all of scipy loaded, so that would add a lot of extra overhead to all programs if we always imported all subpackages.

[Ravaging Care]

This answer given by "alKid" and "falsetru" are not factually correct and rather misleading. The reason lies in the init files of the package. The init of scipy does not import its modules while the init of some other packages like numpy imports its modules. See better answers at the duplicate thread: stackoverflow.com/questions/33379576/…

[SibiCoder]

Always elaborate your answer. In this case, you can say what betterment is available in 1.2.1 version or the bug fixed, etc.

[David Buck]

Welcome to SO. Unfortunately, your answer both doesn't answer the OP's question of "What is wrong with the second approach?" and it is incorrect. Even with scipy 1.4.1 the second approach still results in AttributeError: module 'scipy' has no attribute 'spatial'