Imports in Python
TODO: in which cases is init.py still necessary, even though Python 3.3+ often does not require this anymore?
Sources
These sources are used for this note:
Types of import
There are three types of importing modules in Python:
import X- Now you can refer to things under the
Xnamespace asX.thing - Variant:
import X as xto instead be able to usex.thing
- Now you can refer to things under the
from X import *- import only public object defined in
X Xitself is not defined, so you cannot callX.thing
- import only public object defined in
from X import a, b, c- You can now use
a,bandcin your program
- You can now use
(Ok, a fourth: X = __import__('X'))
Best practice: use syntax 1 with a plain import unless you have a good reason for using the from statement.
Basic terms: module, package, sys.path
- any .py file is a module
- a folder containing
__init__.pyis a package- For Python 3.3+ you don't even need the explicit init file anymore!
- a package without init file is called an (implicit) namespace package
One thing I found confusing, coming from how terminal paths usually work:
sys.pathdoes not care about your current working directory, but only about the path to the script.- "For example, if my shell is currently at the test/ folder and I run python ./packA/subA/subA1.py, then sys.path includes test/packA/subA/ but NOT test/."
sys.pathis shared across all imported modules
Example directory
test/ # root folder
packA/ # package packA
subA/ # subpackage subA
__init__.py
sa1.py
sa2.py
__init__.py
a1.py
a2.py
packB/ # package packB (implicit namespace package)
b1.py
b2.py
math.py
random.py
other.py
start.py
Containing folder of script is not considered to be a package
Subtlety that avoids running __init__.py twice:
Note: If a1.py calls import a2 and we run python a1.py, then test/packA/init.py will NOT be called, even though it seems like a2 is part of the packA package. This is because when Python runs a script (in this case a1.py), its containing folder is not considered a package.
Absolute vs. relative import
- Absolute requires full path from project root
- Explicit relative imports start start with
./- Python 2 supported implicit relative imports but these have been deprecated.
Relative imports cannot be higher than the top-level package!
Note that for relative imports, the dots . can go up only up to (but not including) the directory containing the script run from the command line. Thus, from .. import other is invalid in a1.py. Doing so results in the error ValueError: attempted relative import beyond top-level package.
Absolute imports can be verbose, but at least they are explicit! Things get really tricky if you have scripts in submodules with relative imports. You'll run into trouble when trying to run those submodules directly.
From Python 3 docs:
Note that relative imports are based on the name of the current module. Since the name of the main module is always “main”, modules intended for use as the main module of a Python application must always use absolute imports.
Case studies
These are some common cases where Python importing can get tricky.
Importing and running a submodule directly
Example where I was confused
- Run
python ./src/data/datamodule.py(so working directory is the project root which contains./src/) - which includes
import src.data.datasets.test - ERROR: src is not a module
What happened?
sys.pathonly cares about the script's location i.e./src/data/- But this folder indeed does not contain a folder
src/ - In this case
from data.datasets import testworks if you run the script directly - But this will cause problems when you run another script directly that imports test!
If you instead only run a script from the top-level package, e.g. start.py, then the top-level package is in sys.path.
This means importing all other subpackages from start.py just works.
In this case, sys.path is known ahead of time
EZPZ!
But ideally we are flexible! We want to be able to both ...
- run the module directly on the command line
- import the module from another script
This is not that trivial in Python 3!
This addresses my problem above. Let's say we want to be able to run a2.py directly, but also from start.py. Then we have this problem about the desired import statement in a2.py:
The import statement in a2.py is trickier. When we run start.py directly, sys.path contains test/, so a2.py should call from packA.subA import sa2. However, if we instead run a2.py directly, then sys.path contains test/packA/. Now the import would fail because packA is not a folder inside test/packA/.
Solutions
- Always use absolute imports from the top-level package (in this example
test/).- This way you can always directly run top-level scripts.
- If you want to directly run submodules, you have to run it as an imported module instead of a script
- Change your working directory to the project root
- Run python with the -m flag:
python -m packA.a2
- Explicitly manipulate
sys.pathin submodules
import os, sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))
# now this works, even when a2.py is run directly
from packA.subA import sa2
- Use absolute imports relative to top-level package and then add the top-level package to the
PYTHONPATHvariable- The blog argues against this because it's not portable to permanently add a project root to PYTHONPATH
- You also don't want to manually take care of this each time you want to run This is the solution IDE users use; when running a project you indicate your project root and the IDE temporarily adjusts the PYTHONPATH (e.g. in PyCharm)
- TODO is this where something like setup.py jumps in?
- (Not covered in blog post) install the project root as a package with
pip install -e .to add the project root tosys.path
Import from parent directory
You cannot properly import from parent directories when running a submodule directly! Relative imports also don't help here, because when you run a file directly, that's considered the top-level package and relative imports don't go above the top-level package.
Avoid writing code where this is necessary in the first place.
sys.path manipulation is an option here.
Personally I find that extremely ugly.
Circular imports
Another tricky one that I definitely messed up before.
Let's say we have a module X importing module Y:
# module X
import Y
print("Name:", __name__)
print("X start")
def call():
print("You are calling a function of module X.")
if __name__ == '__main__':
print("X main")
When we call this script, we call the import Y statement first, which executes the code in module Y.
Now let's define module Y with a circular import:
# module Y
import X
print("Name:", __name__)
print("Y start")
X.call()
def call():
print("You are calling a function of module Y.")
if __name__ == '__main__':
print("Y main")
Now if we open an interactive terminal and import X, we run into trouble:
>>> import X
Name: Y
Y start
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\Edwin Wenink\Documents\programming-practice\Python\circular_import\X.py", line 2, in <module>
import Y
File "C:\Users\Edwin Wenink\Documents\programming-practice\Python\circular_import\Y.py", line 6, in <module>
X.call()
AttributeError: module 'X' has no attribute 'call'
This is what happens:
- We import module X
- The first line of module X import Y
- This executes the code in module Y
- Python sees X is already (partially being) imported, so the
import Xstatement inYwill not trigger compilation of the content of X. - This will print "Name: Y" and "Y start"
- Then it will run
X.call() - But the
def call()statement in moduleXhas not been run yet, so we run into this error!
A possible solution could be to run import Y only after declaring the functions that are needed by module Y.
If we do this, we get the following output without error:
>>> import X
X start
Y start
You are calling a function of module X.
Note that importing Y with the above code gives no problems because X is defined before X.call() runs, but only because in this toy example X does currently not use the imported module Y:
>>> import Y
Name: X
X start
Name: Y
Y start
You are calling a function of module X.
In both cases, we see that the main function does not run because neither module is invoked as the main module. The current situation is that importing X throws an error, but importing Y does not. When directly invoking the scripts as the main module, we see the opposite! Calling X:
> python X.py
Name: X
X start
Name: Y
Y start
You are calling a function of module X.
Name: __main__
X start
X main
It took me a bit to understand why this does work now.
The difference is that now we do not start our execution with an import X statement, so that in the first line of Y.py the import X line actually triggers a full run over X.
This means that X.call() is defined after all before it's called in Y!
Another detail we notice now is that Y calls the module X; but when the executing of the main file continues, we are running the __main__ object.
Now let's see what happens when calling Y as the main function:
> python Y.py
Name: Y
Y start
Traceback (most recent call last):
File "Y.py", line 2, in <module>
import X
File "C:\Users\Edwin Wenink\Documents\programming-practice\Python\circular_import\X.py", line 2, in <module>
import Y
File "C:\Users\Edwin Wenink\Documents\programming-practice\Python\circular_import\Y.py", line 6, in <module>
X.call()
AttributeError: module 'X' has no attribute 'call'
Now, executing Y triggers import X.
This in turn, will trigger import Y.
Because Y is not in the list of modules yet (we haven't run import Y), Y will be compiled and we'll encounter X.call() before this function is defined.
TLDR; avoid circular imports!
Things get even more complicated when Y is also called from X.
Even when you can postpone some imports to quickly fix the situation, it's better to avoid this situation altogether by refactoring your code.
Otherwise, your code will break if a colleague wonders why there's an import statement at the bottom of the file and moves it up.
When do we need init.py?
I'm working in a project with the following structure:
main.py
src/
caseloader.py
...
pipeline.py
utils.py
logs/
...
My main.py file is importing from src.pipeline import run_pipeline, but this fails with ModuleNotFoundError: No module named 'src.pipeline'.
Adding an empty __init__.py to src fixes this.
TODO: Why is this?