Imports in Python

TODO: in which cases is init.py still necessary, even though Python 3.3+ often does not require this anymore?

Sources

These sources are used for this note:

Types of import

There are three types of importing modules in Python:

  1. import X
    • Now you can refer to things under the X namespace as X.thing
    • Variant: import X as x to instead be able to use x.thing
  2. from X import *
    • import only public object defined in X
    • X itself is not defined, so you cannot call X.thing
  3. from X import a, b, c
    • You can now use a, b and c in your program

(Ok, a fourth: X = __import__('X'))

Best practice: use syntax 1 with a plain import unless you have a good reason for using the from statement.

Basic terms: module, package, sys.path

  • any .py file is a module
  • a folder containing __init__.py is a package
    • For Python 3.3+ you don't even need the explicit init file anymore!
    • a package without init file is called an (implicit) namespace package

One thing I found confusing, coming from how terminal paths usually work:

  • sys.path does not care about your current working directory, but only about the path to the script.
  • "For example, if my shell is currently at the test/ folder and I run python ./packA/subA/subA1.py, then sys.path includes test/packA/subA/ but NOT test/."
  • sys.path is shared across all imported modules

Example directory

test/                      # root folder
    packA/                 # package packA
        subA/              # subpackage subA
            __init__.py
            sa1.py
            sa2.py
        __init__.py
        a1.py
        a2.py
    packB/                 # package packB (implicit namespace package)
        b1.py
        b2.py
    math.py
    random.py
    other.py
    start.py

Containing folder of script is not considered to be a package

Subtlety that avoids running __init__.py twice:

Note: If a1.py calls import a2 and we run python a1.py, then test/packA/init.py will NOT be called, even though it seems like a2 is part of the packA package. This is because when Python runs a script (in this case a1.py), its containing folder is not considered a package.

Absolute vs. relative import

  • Absolute requires full path from project root
  • Explicit relative imports start start with ./
    • Python 2 supported implicit relative imports but these have been deprecated.

Relative imports cannot be higher than the top-level package!

Note that for relative imports, the dots . can go up only up to (but not including) the directory containing the script run from the command line. Thus, from .. import other is invalid in a1.py. Doing so results in the error ValueError: attempted relative import beyond top-level package.

Absolute imports can be verbose, but at least they are explicit! Things get really tricky if you have scripts in submodules with relative imports. You'll run into trouble when trying to run those submodules directly.

From Python 3 docs:

Note that relative imports are based on the name of the current module. Since the name of the main module is always “main”, modules intended for use as the main module of a Python application must always use absolute imports.

Case studies

These are some common cases where Python importing can get tricky.

Importing and running a submodule directly

Example where I was confused

  • Run python ./src/data/datamodule.py (so working directory is the project root which contains ./src/)
  • which includes import src.data.datasets.test
  • ERROR: src is not a module

What happened?

  • sys.path only cares about the script's location i.e. /src/data/
  • But this folder indeed does not contain a folder src/
  • In this case from data.datasets import test works if you run the script directly
  • But this will cause problems when you run another script directly that imports test!

If you instead only run a script from the top-level package, e.g. start.py, then the top-level package is in sys.path. This means importing all other subpackages from start.py just works. In this case, sys.path is known ahead of time EZPZ!

But ideally we are flexible! We want to be able to both ...

  • run the module directly on the command line
  • import the module from another script

This is not that trivial in Python 3!

This addresses my problem above. Let's say we want to be able to run a2.py directly, but also from start.py. Then we have this problem about the desired import statement in a2.py:

The import statement in a2.py is trickier. When we run start.py directly, sys.path contains test/, so a2.py should call from packA.subA import sa2. However, if we instead run a2.py directly, then sys.path contains test/packA/. Now the import would fail because packA is not a folder inside test/packA/.

Solutions

  1. Always use absolute imports from the top-level package (in this example test/).
    • This way you can always directly run top-level scripts.
    • If you want to directly run submodules, you have to run it as an imported module instead of a script
      1. Change your working directory to the project root
      2. Run python with the -m flag: python -m packA.a2
  2. Explicitly manipulate sys.path in submodules
 import os, sys
 sys.path.append(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))

 # now this works, even when a2.py is run directly
 from packA.subA import sa2
  1. Use absolute imports relative to top-level package and then add the top-level package to the PYTHONPATH variable
    • The blog argues against this because it's not portable to permanently add a project root to PYTHONPATH
    • You also don't want to manually take care of this each time you want to run This is the solution IDE users use; when running a project you indicate your project root and the IDE temporarily adjusts the PYTHONPATH (e.g. in PyCharm)
    • TODO is this where something like setup.py jumps in?
  2. (Not covered in blog post) install the project root as a package with pip install -e . to add the project root to sys.path

Import from parent directory

You cannot properly import from parent directories when running a submodule directly! Relative imports also don't help here, because when you run a file directly, that's considered the top-level package and relative imports don't go above the top-level package.

Avoid writing code where this is necessary in the first place.

sys.path manipulation is an option here. Personally I find that extremely ugly.

Circular imports

Another tricky one that I definitely messed up before.

Let's say we have a module X importing module Y:

# module X
import Y

print("Name:", __name__)
print("X start")


def call():
    print("You are calling a function of module X.")


if __name__ == '__main__':
    print("X main")

When we call this script, we call the import Y statement first, which executes the code in module Y. Now let's define module Y with a circular import:

# module Y
import X

print("Name:", __name__)
print("Y start")

X.call()


def call():
    print("You are calling a function of module Y.")


if __name__ == '__main__':
    print("Y main")

Now if we open an interactive terminal and import X, we run into trouble:

>>> import X
Name: Y
Y start
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Edwin Wenink\Documents\programming-practice\Python\circular_import\X.py", line 2, in <module>
    import Y
  File "C:\Users\Edwin Wenink\Documents\programming-practice\Python\circular_import\Y.py", line 6, in <module>
    X.call()
AttributeError: module 'X' has no attribute 'call'

This is what happens:

  1. We import module X
  2. The first line of module X import Y
  3. This executes the code in module Y
  4. Python sees X is already (partially being) imported, so the import X statement in Y will not trigger compilation of the content of X.
  5. This will print "Name: Y" and "Y start"
  6. Then it will run X.call()
  7. But the def call() statement in module X has not been run yet, so we run into this error!

A possible solution could be to run import Y only after declaring the functions that are needed by module Y. If we do this, we get the following output without error:

>>> import X
X start
Y start
You are calling a function of module X.

Note that importing Y with the above code gives no problems because X is defined before X.call() runs, but only because in this toy example X does currently not use the imported module Y:

>>> import Y
Name: X
X start
Name: Y
Y start
You are calling a function of module X.

In both cases, we see that the main function does not run because neither module is invoked as the main module. The current situation is that importing X throws an error, but importing Y does not. When directly invoking the scripts as the main module, we see the opposite! Calling X:

> python X.py
Name: X
X start
Name: Y
Y start
You are calling a function of module X.
Name: __main__
X start
X main

It took me a bit to understand why this does work now. The difference is that now we do not start our execution with an import X statement, so that in the first line of Y.py the import X line actually triggers a full run over X. This means that X.call() is defined after all before it's called in Y! Another detail we notice now is that Y calls the module X; but when the executing of the main file continues, we are running the __main__ object.

Now let's see what happens when calling Y as the main function:

> python Y.py
Name: Y
Y start
Traceback (most recent call last):
  File "Y.py", line 2, in <module>
    import X
  File "C:\Users\Edwin Wenink\Documents\programming-practice\Python\circular_import\X.py", line 2, in <module>
    import Y
  File "C:\Users\Edwin Wenink\Documents\programming-practice\Python\circular_import\Y.py", line 6, in <module>
    X.call()
AttributeError: module 'X' has no attribute 'call'

Now, executing Y triggers import X. This in turn, will trigger import Y. Because Y is not in the list of modules yet (we haven't run import Y), Y will be compiled and we'll encounter X.call() before this function is defined.

TLDR; avoid circular imports!

Things get even more complicated when Y is also called from X. Even when you can postpone some imports to quickly fix the situation, it's better to avoid this situation altogether by refactoring your code. Otherwise, your code will break if a colleague wonders why there's an import statement at the bottom of the file and moves it up.

When do we need init.py?

I'm working in a project with the following structure:

main.py
src/
    caseloader.py
    ...
    pipeline.py
    utils.py
logs/
...

My main.py file is importing from src.pipeline import run_pipeline, but this fails with ModuleNotFoundError: No module named 'src.pipeline'. Adding an empty __init__.py to src fixes this.

TODO: Why is this?