Circular import dependency in Python

Question

Let's say I have the following directory structure:

a\
    __init__.py
    b\
        __init__.py
        c\
            __init__.py
            c_file.py
        d\
            __init__.py
            d_file.py

In the a package's __init__.py, the c package is imported. But c_file.py imports a.b.d.

The program fails, saying b doesn't exist when c_file.py tries to import a.b.d. (And it really doesn't exist, because we were in the middle of importing it.)

How can this problem be remedied?

Solution 1

If a depends on c and c depends on a, aren't they actually the same unit then?

You should really examine why you have split a and c into two packages, because either you have some code you should split off into another package (to make them both depend on that new package, but not each other), or you should merge them into one package.

Solution 2

You may defer the import, for example in a/__init__.py:

def my_function():
    from a.b.c import Blah
    return Blah()

that is, defer the import until it is really needed. However, I would also have a close look at my package definitions/uses, as a cyclic dependency like the one pointed out might indicate a design problem.

Solution 3

I've wondered this a couple times (usually while dealing with models that need to know about each other). The simple solution is just to import the whole module, then reference the thing that you need.

So instead of doing

from models import Student

in one, and

from models import Classroom

in the other, just do

import models

in one of them, then call models.Classroom when you need it.

Solution 4

Circular Dependencies due to Type Hints

With type hints, there are more opportunities for creating circular imports. Fortunately, there is a solution using the special constant: typing.TYPE_CHECKING.

The following example defines a Vertex class and an Edge class. An edge is defined by two vertices and a vertex maintains a list of the adjacent edges to which it belongs.

Without Type Hints, No Error

File: vertex.py

class Vertex:
    def __init__(self, label):
        self.label = label
        self.adjacency_list = []

File: edge.py

class Edge:
    def __init__(self, v1, v2):
        self.v1 = v1
        self.v2 = v2

Type Hints Cause ImportError

ImportError: cannot import name 'Edge' from partially initialized module 'edge' (most likely due to a circular import)

File: vertex.py

from typing import List
from edge import Edge


class Vertex:
    def __init__(self, label: str):
        self.label = label
        self.adjacency_list: List[Edge] = []

File: edge.py

from vertex import Vertex


class Edge:
    def __init__(self, v1: Vertex, v2: Vertex):
        self.v1 = v1
        self.v2 = v2

Solution using TYPE_CHECKING

File: vertex.py

from typing import List, TYPE_CHECKING

if TYPE_CHECKING:
    from edge import Edge


class Vertex:
    def __init__(self, label: str):
        self.label = label
        self.adjacency_list: List['Edge'] = []

File: edge.py

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from vertex import Vertex


class Edge:
    def __init__(self, v1: 'Vertex', v2: 'Vertex'):
        self.v1 = v1
        self.v2 = v2

Quoted vs. Unquoted Type Hints

In versions of Python prior to 3.10, conditionally imported types must be enclosed in quotes, making them “forward references”, which hides them from the interpreter runtime.

In Python 3.7, 3.8, and 3.9, a workaround is to use the following special import.

from __future__ import annotations

This enables using unquoted type hints combined with conditional imports.

Python 3.10 (See PEP 563 -- Postponed Evaluation of Annotations)

In Python 3.10, function and variable annotations will no longer be evaluated at definition time. Instead, a string form will be preserved in the respective annotations dictionary. Static type checkers will see no difference in behavior, whereas tools using annotations at runtime will have to perform postponed evaluation.

The string form is obtained from the AST during the compilation step, which means that the string form might not preserve the exact formatting of the source. Note: if an annotation was a string literal already, it will still be wrapped in a string.

View more solutions

Comments

[Matthew Lund]

Yes, they could be considered the same package. But if this results in a massively huge file then it's impractical. I agree that frequently, circular dependencies mean the design should be thought through again. But there ARE some design patterns where it's appropriate (and where merging the files together would result in a huge file) so I think it's dogmatic to say that the packages should either be combined or the design should be re-evaluated.

[Antimony]

modifying global module attributes in a different file like that will quickly lead to a nightmare

[Jason Polites]

Sometimes circular references are truly unavoidable. This is the only approach that works for me in these circumstances.

[Mr_and_Mrs_D]

Wouldn't this add a lot of overhead in every call of foo ?

[Dirk]

@Mr_and_Mrs_D - only moderately. Python keeps all imported modules in a global cache (sys.modules), so once a module has been loaded, it won't be loaded again. The code might involve a name look up on each call to my_function, but so does code, which references symbols via qualified names (e.g., import foo; foo.frobnicate())

[Richard J]

of all the possible solutions here, this is the only one which worked for me. There are absolutely circumstances where a circular reference is "the best" solution - particularly when what you are doing is splitting a set of model objects across multiple files for constraining file sizes.

[Julie in Austin]

Sometimes circular references are precisely the correct way to model the problem. The notion that circular dependencies are somehow an indication of poor design seems to be more a reflection on Python as a language rather than a legitimate design point.

[TomSawyer]

Does import in middle of the file like this is considered as bad practice?

[Dirk]

@TomSawyer - I would not consider this a bad practice per se (in particular, since it might be a plain necessity sometimes). I would, however, try to solve the problem by restructuring my module dependencies first, before giving up and using this solution, simply because I myself find it less readable. What's actually considered a bad practice is doing from xxx import * in a local scope. But then, import * is frowned upon even on a module level, and a syntax error anyway in local scopes in "modern" Python, which renders the question mood.

[R OMS]

Can you show use what models.py looks like? I don't want to put all the class definitions in one file. I want to create a models.py that imports each class from it's own file. I need to see an example file structure.

[zachaysan]

It doesn't need to be one file @ROMS models can be a directory that has an __init__.py file that does the importing from models.classroom.