How to organize a Python Project?

Question

I'm new to Python and I'm starting a mini Project, but I have some doubts on how to organize the folders in the "Python Way".

I'm using PyDev in my Development Environment, and when I create a new project a folder is created called src

+ src

Now, in the PyDev, I can create Pydev Module and PyDev Package

I need to organize my Project in the following way:

+ Indicators
    - Moving_averages.py
    - Stochastics.py
+ Strategies
    - Moving_averages_cross.py
- example.py

How can I organize this in terms of Modules and Packages? What is the meaning of Modules and Packages?

Solution 1

A Package is basically a folder with __init__.py file under it and usually some Modules, where Module is a *.py file. It has to do with import mainly. If you add __init__.py to Indicators you can use:

from Indicators.Stochastics import *

or

from Indicators import Stochastics

By the way, I would recommend to keep module/package names lowercase. It does not affect functionality but it's more "pythonic".

Solution 2

From a file system perspective, a module is a file ending with .py and a package is a folder containing modules and (nested) packages again. Python recognizes a folder as a package if it contains a __init__.py file.

A file structure like that

some/
    __init__.py
    foofoo.py
    thing/
        __init__.py
        barbar.py

defines the package some, which has a module foofoo and a nested package thing, which again has a module barbar. However, when using packages and modules, you don't really distinguish these two types:

import some

some.dothis() # dothis is defined in 'some/__init__.py'

import some.foofoo # <- module
import some.thing # <- package

Please follow PEP8 when selecting naming your packages/modules (i.e. use lower-case names).

Solution 3

See python-package-template

Directory structure

    .
    |-- bin
    |   `-- my_program
    |-- docs
    |   `-- doc.txt
    |-- my_program
    |   |-- data
    |   |   `-- some_data.html
    |   |-- __init__.py
    |   |-- submodule
    |   |   `-- __init__.py
    |   |-- helpers.py
    |-- tests
    |   |-- __init__.py
    |   |-- test_helpers.py
    |-- Makefile
    |-- CHANGES.txt
    |-- LICENSE.txt
    |-- README.md
    |-- requirements-dev.txt
    |-- requirements.txt
    `-- setup.py

cat Makefile

    PYTHON=`which python`
    NAME=`python setup.py --name`


    all: check test source deb

    init:
        pip install -r requirements.txt --use-mirrors

    dist: source deb

    source:
        $(PYTHON) setup.py sdist

    deb:
        $(PYTHON) setup.py --command-packages=stdeb.command bdist_deb

    rpm:
        $(PYTHON) setup.py bdist_rpm --post-install=rpm/postinstall --pre-uninstall=rpm/preuninstall

    test:
        unit2 discover -s tests -t .
        python -mpytest weasyprint

    check:
        find . -name \*.py | grep -v "^test_" | xargs pylint --errors-only --reports=n
        # pep8
        # pyntch
        # pyflakes
        # pychecker
        # pymetrics

    clean:
        $(PYTHON) setup.py clean
        rm -rf build/ MANIFEST dist build my_program.egg-info deb_dist
        find . -name '*.pyc' -delete

Solution 4

You might want to check out the modern-package-template libary. It provides a way to setup a really nice basic layout for a project that walks you through a few questions and tries to help you get something that's able to be distributed fairly easily.

http://pypi.python.org/pypi/modern-package-template

Solution 5

Before deciding on a project structure, it's good to ask yourself what the purpose of the project is going to be. Is this going to be one off analysis? A toy concept you want to investigate? A full blown project you intend to distribute? The amount of effort you want to put into structuring your project will be different.

• If it's a one off analysis, I like to use ipython notebooks. The notebook will capture the flow of your thoughts, and you can add notes in markup to your code for later reference.
• If it's a toy concept you want to investigate, I find a simple, quick approach to work best. You want to be able to quickly implement your concept to discover if it's even feasible and thus worth spending more time on it. Part of Python's philosophy is 'Don’t try for perfection because “good enough” is often just that.' You can always come back later and structure your project in a way that follows best software engineering practices.

• If you want to structure your project so you can later distribute it, and so that it scales to many modules I recommend the following structure:

  projectname
   ├── MANIFEST.in
   ├── setup.py
   ├── README
   ├── .gitignore
   ├── .git
   ├── projectname_env
   └── projectname
       ├── __init__.py
       ├── subpackageone
       │   ├── __init__.py
       │   ├── second_module.py
       │   ├── tests
       │   │   └── test_second_module.py
       │   └── models
       │       └── model1
       ├── first_module.py   
       └── tests
           └── test_second_module.py
  

The detailed reasons why I like this structure are in my blog post, but the basic gist is that the hierarchically lower level projectname directory contains your actual project. Alongside it are all the tools that help manage (git) and package (setup.py, MANIFEST.in) it.

View more solutions

Comments

[LiorK]

I can recommend this link, out of the official Python Tutorial. Is it pretty much clear and straight forward.

[Bayleef]

A very recent helpful post: Starting a python project the right way

[Alfian Nahar]

hi, Vitaly! I've just come across your project. Are you still maintaining it? Thanks!

[Jay Wong]

Can I call the function from foofoo.py if I only import some? I am confused about the __init__.py, what should I put there?

[Dimitrios Mistriotis]

Ended up with a 404

[Toothpick Anemone]

The link https://github.com/vital-fadeev/python-package-template is no longer valid. Where can this python-package-template be found?

[buhtz]

How would you run your project? In the root folder doing python3 -m projectname?

[Jawad Mehmood]

@JayWong It's not compulsory to put something in __init__.py, it helps python accept the current folder as a module and *.py files as packages. Whenever you import a module, the __init__.py file runs and executes whatever's in it.