Attempted relative import beyond toplevel package

Question

Here is my folder structure:

Mopy/ # no init.py !
   bash/
     __init__.py
     bash.py # <--- Edit: yep there is such a module too
     bass.py
     bosh/
       __init__.py # contains from .. import bass
       bsa_files.py
     ...
   test_bash\
     __init__.py # code below
     test_bosh\
       __init__.py
       test_bsa_files.py

In test_bash\__init__.py I have:

import sys
from os.path import dirname, abspath, join, sep
mopy = dirname(dirname(abspath(__file__)))
assert mopy.split(sep)[-1].lower() == 'mopy'
sys.path.append(mopy)
print 'Mopy folder appended to path: ', mopy

while in test_bsa_files.py:

import unittest
from unittest import TestCase

import bosh

class TestBSAHeader(TestCase):
    def test_read_header(self):
        bosh.bsa_files.Header.read_header()

if __name__ == '__main__':
    unittest.main()

Now when I issue:

python.exe "C:\_\JetBrains\PyCharm 2016.2.2\helpers\pycharm\utrunner.py" C:\path\to\Mopy\test_bash\test_bosh\test_bsa_files.py true

I get:

Traceback (most recent call last):
  File "C:\_\JetBrains\PyCharm 2016.2.2\helpers\pycharm\utrunner.py", line 124, in <module>
    modules = [loadSource(a[0])]
  File "C:\_\JetBrains\PyCharm 2016.2.2\helpers\pycharm\utrunner.py", line 43, in loadSource
    module = imp.load_source(moduleName, fileName)
  File "C:\Dropbox\eclipse_workspaces\python\wrye-bash\Mopy\test_bash\test_bosh\test_bsa_files.py", line 4, in <module>
    import bosh
  File "C:\Dropbox\eclipse_workspaces\python\wrye-bash\Mopy\bash\bosh\__init__.py", line 50, in <module>
    from .. import bass
ValueError: Attempted relative import beyond toplevel package

Since 'Mopy" is in the sys.path and bosh\__init__.py is correctly resolved why it complains about relative import above the top level package ? Which is the top level package ?

Incidentally this is my attempt to add tests to a legacy project - had asked in Python test package layout but was closed as a duplicate of Where do the Python unit tests go?. Comments on my current test package layout are much appreciated !

---

Well the answer below does not work in my case:

The module bash.py is the entry point to the application containing:

if __name__ == '__main__':
    main()

When I use import bash.bosh or from bash import bosh I get:

C:\_\Python27\python.exe "C:\_\JetBrains\PyCharm 2016.2.2\helpers\pycharm\utrunner.py" C:\Dropbox\eclipse_workspaces\python\wrye-bash\Mopy\test_bash\test_bosh\test_bsa_files.py true
Testing started at 3:45 PM ...
usage: utrunner.py [-h] [-o OBLIVIONPATH] [-p PERSONALPATH] [-u USERPATH]
                   [-l LOCALAPPDATAPATH] [-b] [-r] [-f FILENAME] [-q] [-i]
                   [-I] [-g GAMENAME] [-d] [-C] [-P] [--no-uac] [--uac]
                   [--bashmon] [-L LANGUAGE]
utrunner.py: error: unrecognized arguments: C:\Dropbox\eclipse_workspaces\python\wrye-bash\Mopy\test_bash\test_bosh\test_bsa_files.py true

Process finished with exit code 2

This usage message is from the main() in bash.

Solution

TLDR: Do

import bash.bosh

or

from bash import bosh

Avoid modifying sys.path, as this duplicates modules.

---

When you do

import bosh

it will import the module bosh. This means Mopy/bash is in your sys.path, python finds the file bosh there, and imports it. The module is now globally known by the name bosh. Whether bosh is itself a module or package doesn't matter for this, it only changes whether bosh.py or bosh/__init__.py is used.

Now, when bosh tries to do

from .. import bass

this is not a file system operation ("one directory up, file bass") but a module name operation. It means "one package level up, module bass". bosh wasn't imported from its package, but on its own, though. So going up one package is not possible - you end up at the package '', which is not valid.

Let's look at what happens when you do

import bash.bosh

instead. First, the package bash is imported. Then, bosh is imported as a module of that package - it is globally know as bash.bosh, even if you used from bash import bosh.

When bosh does

from .. import bass

that one works now: going one level up from bash.bosh gets you to bash. From there, bass is imported as bash.bass.

Comments

[MisterMiyagi]

Do you have any unguarded code in bash.py working with sys.argv or argparse? The if __name__ == '__main__': will not trigger on import bash or any of its variants.

[Mr_and_Mrs_D]

@MisterMiyagi: Yep you got it - this line here parses the cli: github.com/wrye-bash/wrye-bash/blob/…

[MisterMiyagi]

You should move that one to if __name__ ..., though it will not fix the problem. I've updated the answer, giving the package higher priority over its modules.

[Mr_and_Mrs_D]

I had tried that btw and I had got the message but I did not mention (along with countless other such hit in the dark)

[MisterMiyagi]

@Mr_and_Mrs_D You have duplicate names in your file hierarchy and think that might not be worth mentioning? oO Let me see if I can figure that one out...

[MisterMiyagi]

@Mr_and_Mrs_D Added a piece for changing the preference of module vs package code. Give it a shot.

[Mr_and_Mrs_D]

Nope :( exact same behavior as with append - still I see the usage() from bash.main() (hahaha, I mean, and imagine we even have a custom loader - not the simplest package structure in the world...)

[Mr_and_Mrs_D]

Wait wait - let's move this inside test_bsa_files - edit: yep that worked !!! I did it cause I noticed that the debug print ('print 'Mopy folder appended to path: ', mopy` was not seen in the last output - the one with usage()). But why is not the test_bash\__init__.py run in this case ?

[MisterMiyagi]

@Mr_and_Mrs_D When you are running just test_bosh\test_bar.py directly, that has the same issue - never going through its parent package test_bosh.

[Mohammad Shahid Siddiqui]

You can repeat the depth of modules and child modules. E.g. import ProjectDir.sibling2.submodule.subsubmodule ...