What is the best project structure for a Python application?
Solution 1
Doesn't too much matter. Whatever makes you happy will work. There aren't a lot of silly rules because Python projects can be simple.
-
/scripts
or/bin
for that kind of command-line interface stuff -
/tests
for your tests -
/lib
for your C-language libraries -
/doc
for most documentation -
/apidoc
for the Epydoc-generated API docs.
And the top-level directory can contain README's, Config's and whatnot.
The hard choice is whether or not to use a /src
tree. Python doesn't have a distinction between /src
, /lib
, and /bin
like Java or C has.
Since a top-level /src
directory is seen by some as meaningless, your top-level directory can be the top-level architecture of your application.
/foo
/bar
/baz
I recommend putting all of this under the "name-of-my-product" directory. So, if you're writing an application named quux
, the directory that contains all this stuff is named /quux
.
Another project's PYTHONPATH
, then, can include /path/to/quux/foo
to reuse the QUUX.foo
module.
In my case, since I use Komodo Edit, my IDE cuft is a single .KPF file. I actually put that in the top-level /quux
directory, and omit adding it to SVN.
Solution 2
According to Jean-Paul Calderone's Filesystem structure of a Python project:
Project/
|-- bin/
| |-- project
|
|-- project/
| |-- test/
| | |-- __init__.py
| | |-- test_main.py
| |
| |-- __init__.py
| |-- main.py
|
|-- setup.py
|-- README
Solution 3
This blog post by Jean-Paul Calderone is commonly given as an answer in #python on Freenode.
Filesystem structure of a Python project
Do:
- name the directory something related to your project. For example, if your project is named "Twisted", name the top-level directory for its source files
Twisted
. When you do releases, you should include a version number suffix:Twisted-2.5
.- create a directory
Twisted/bin
and put your executables there, if you have any. Don't give them a.py
extension, even if they are Python source files. Don't put any code in them except an import of and call to a main function defined somewhere else in your projects. (Slight wrinkle: since on Windows, the interpreter is selected by the file extension, your Windows users actually do want the .py extension. So, when you package for Windows, you may want to add it. Unfortunately there's no easy distutils trick that I know of to automate this process. Considering that on POSIX the .py extension is a only a wart, whereas on Windows the lack is an actual bug, if your userbase includes Windows users, you may want to opt to just have the .py extension everywhere.)- If your project is expressable as a single Python source file, then put it into the directory and name it something related to your project. For example,
Twisted/twisted.py
. If you need multiple source files, create a package instead (Twisted/twisted/
, with an emptyTwisted/twisted/__init__.py
) and place your source files in it. For example,Twisted/twisted/internet.py
.- put your unit tests in a sub-package of your package (note - this means that the single Python source file option above was a trick - you always need at least one other file for your unit tests). For example,
Twisted/twisted/test/
. Of course, make it a package withTwisted/twisted/test/__init__.py
. Place tests in files likeTwisted/twisted/test/test_internet.py
.- add
Twisted/README
andTwisted/setup.py
to explain and install your software, respectively, if you're feeling nice.Don't:
- put your source in a directory called
src
orlib
. This makes it hard to run without installing.- put your tests outside of your Python package. This makes it hard to run the tests against an installed version.
- create a package that only has a
__init__.py
and then put all your code into__init__.py
. Just make a module instead of a package, it's simpler.- try to come up with magical hacks to make Python able to import your module or package without having the user add the directory containing it to their import path (either via PYTHONPATH or some other mechanism). You will not correctly handle all cases and users will get angry at you when your software doesn't work in their environment.
Solution 4
Check out Open Sourcing a Python Project the Right Way.
Let me excerpt the project layout part of that excellent article:
When setting up a project, the layout (or directory structure) is important to get right. A sensible layout means that potential contributors don't have to spend forever hunting for a piece of code; file locations are intuitive. Since we're dealing with an existing project, it means you'll probably need to move some stuff around.
Let's start at the top. Most projects have a number of top-level files (like setup.py, README.md, requirements.txt, etc). There are then three directories that every project should have:
- A docs directory containing project documentation
- A directory named with the project's name which stores the actual Python package
- A test directory in one of two places
- Under the package directory containing test code and resources
- As a stand-alone top level directory To get a better sense of how your files should be organized, here's a simplified snapshot of the layout for one of my projects, sandman:
$ pwd ~/code/sandman $ tree . |- LICENSE |- README.md |- TODO.md |- docs | |-- conf.py | |-- generated | |-- index.rst | |-- installation.rst | |-- modules.rst | |-- quickstart.rst | |-- sandman.rst |- requirements.txt |- sandman | |-- __init__.py | |-- exception.py | |-- model.py | |-- sandman.py | |-- test | |-- models.py | |-- test_sandman.py |- setup.py
As you can see, there are some top level files, a docs directory (generated is an empty directory where sphinx will put the generated documentation), a sandman directory, and a test directory under sandman.
Solution 5
The "Python Packaging Authority" has a sampleproject:
https://github.com/pypa/sampleproject
It is a sample project that exists as an aid to the Python Packaging User Guide's Tutorial on Packaging and Distributing Projects.
kbluck
Updated on January 31, 2022Comments
-
kbluck about 2 years
Imagine that you want to develop a non-trivial end-user desktop (not web) application in Python. What is the best way to structure the project's folder hierarchy?
Desirable features are ease of maintenance, IDE-friendliness, suitability for source control branching/merging, and easy generation of install packages.
In particular:
- Where do you put the source?
- Where do you put application startup scripts?
- Where do you put the IDE project cruft?
- Where do you put the unit/acceptance tests?
- Where do you put non-Python data such as config files?
- Where do you put non-Python sources such as C++ for pyd/so binary extension modules?
-
Lance Rushing over 14 yearsAny open source python projects you would recommend emulating their directory structure?
-
user1066101 over 14 yearsLook at Django for a good example.
-
Charles Duffy over 13 yearsI don't tend to consider Django a good example -- playing tricks with sys.path is an instant DQ in my book.
-
user1066101 over 13 years@Charles Duffy: "tricks with sys.path"? Can you provide a link or quote?
-
Raffi Khatchadourian over 12 yearsI am an advocate of the src directory.
-
Jonathan Hartley over 12 yearsre "tricks": Django adds the parent of the root project folder to the sys.path, so that modules can be imported as either "from project.app.module import klass" or "from app.module import klass".
-
Jack O'Connor about 11 yearsThis was exactly what I needed. "DONT try to come up with magical hacks to make Python able to import your module or package without having the user add the directory containing it to their import path." Good to know!
-
saccharine over 10 yearsAfter blithely ignoring this I eventually had to shuffle my stuff to look like this in order to make it easy to make rpms out of. Highly recommend everyone use this setup by default unless you have a really good reason not to.
-
ThorSummoner almost 10 yearshow does executable file in the bin folder reference the project module? (I don't think python syntax allows
../
in an include statement) -
matanc1 over 9 yearsThanks for a great answer! You clarified many things for me! I just have one question: Can eggs be nested?
-
László Papp over 9 yearsThing is, this does not mention the important doc part of a project where to place it.
-
KT. over 9 yearsNo, you can't "nest" eggs in the sense of storing .egg files within other .egg files and hoping this would be of much use [unless you're up to something really weird]. What you can do, though, is create "virtual" eggs - empty packages that do not provide any useful code, but list other packages in their dependency lists. This way, when a user attempts to install such a package he will recursively install many dependent eggs.
-
Ad N over 9 years@ThorSummoner That is actually a very good question to me, appart from tweaking the PYTHON_PATH I do not see how to do it. It seems to me that this structure makes running the project startup script unnecessarily complicated (putting the 'project' file directly under project/ removes the need to tweak PYTHON_PATH).
-
ThorSummoner over 9 yearsActually, Python does support going up directories; See stackoverflow.com/a/1054281/1695680.
-
Yonatan over 9 years@ThorSummoner That only works when remaining inside a single package. For the relative import to work here you'd need an
__init__.py
file in both thebin
folder and theProject
top folder. -
Notre over 9 yearsI do this, but more so: I have a toplevel Makefile with an 'env' target that automates 'virtualenv env ; ./env/bin/pip install -r requirements.txt ; ./env/bin/python setup.py develop', and also usually a 'test' target that depends on env and also installs test dependencies and then runs py.test.
-
Kroltan over 9 years@ThorSummoner Simple. You install the package! (
pip install -e /path/to/Project
) -
jeremyjjbrown over 9 yearsIt would be awesome if someone would zip up a sample of this layout with hello.py and hello-test.py and make it available for us newbs.
-
Peter Ehrlich about 9 yearsConfused about "put your source in a directory called src or lib. This makes it hard to run without installing.". What would be installed? Is it the dir name that causes the issue, or the fact that it is a subdir at all?
-
Tomas Tomecek about 9 yearshaving folder
/bin
with executables sounds like a very bad practice to me: you have problems with importing as mentioned above and have to fiddle withPYTHONPATH
; I prefer using entry_points -
eric almost 9 yearsYep. I try to be "Pythonic" about it: explicit is better than implicit.. Directory heirarchies are read/inspected more than they are written. Etc..
-
Admin over 8 yearsWhat is the naming convention for project Directory for python PascalCase or this-case?
-
Bloke about 8 years@Kroltan Could you please explain what happens after
pip install
ing the package that allows you to nicely run an executable file from thebin
folder which nicely references the project module? -
Kroltan about 8 years@Bloke The core is the
-e
flag, which installs the package as an editable package, that is, installs it as links to the actual project folder. The executable can then justimport project
to have access to the module. -
cmyr about 8 years@KT can you elaborate a bit about how you handle generated data? In particular, how do you (in code) distinguish between development and deployment? I imagine you have some
base_data_location
variable, but how do you set it appropriately? -
KT. about 8 yearsI presume you are speaking about "runtime data" - something people would often put under /var/packagename or ~/.packagename/var, or whatnot. Most of the time those choices are sufficient as a default that your users don't care to change. If you want to allow this behaviour to be tuned, options are rather abundant and I do not think there is a single fits-all best practice. Typical choices: a) ~/.packagename/configfile, b) export MY_PACKAGE_CONFIG=/path/to/configfile c) command-line options or function parameters d) combination of those.
-
KT. about 8 yearsNote that it is quite usual to have a singleton Config class somewhere, which handles your favourite config loading logic for you and perhaps even lets the user modify the settings at runtime. In general, though, I think this is an issue worth a separate question (which might have been asked before somewhere here).
-
endolith almost 7 years"Some people will assert that you should distribute your tests within your module itself – I disagree. It often increases complexity for your users; many test suites often require additional dependencies and runtime contexts." python-guide-pt-br.readthedocs.io/en/latest/writing/structure/…
-
Yongwei Wu almost 7 yearsOh, I love this trick and am using it now. I want to put the shared module in another directory, and I do not want to install module system-wide, nor do I want to ask people to modify PYTHONPATH manually. Unless people propose something better, I think this is actually the cleanest way to go.
-
MacSanhe over 6 yearsI recently create a tool can help you initiate a python project from scratch. by typing some info like: package_name, github_username, license, python version you want to support. And having virtualenv, test, code coverage, doc, deploy doc, publish to pypi, multi python version test (tox) out of the box! And it compatible with WINDOWS / MACOS / LINUX. Check this out: pypi.python.org/pypi/pygitrepo
-
Collin Bell over 6 yearscuft -> cruft* .
-
user1071847 about 6 yearsRe Django's tricks, I believe they removed that as of version 1.4. (See this discussion.)
-
GoodDeeds almost 6 years@JackO'Connor I am confused. How do you do an import to the main function defined inside
Twisted/twisted
from the executable inTwisted/bin
without modifying paths? -
Jack O'Connor almost 6 years@GoodDeeds I think the standard way is to have the user install the library, either globally or in some virtualenv that they're using, so that the import works. I do sometimes like to include a bash script that sets the necessary PYTHONPATH and executes the code locally, to help with development, but I'm not sure what other people do.
-
trk over 5 yearsThis thread is quite old, but I can shed some light on the matter of putting the executable
project
inbin
. Apparently it's relevant to Linux systems where the executable is meant to usually be a shell script (with the same name as your Python project). More info here. -
Nick T almost 5 years"This makes it hard to run without installing." -- that's the point
-
Maggyero over 4 yearsI find it ironical that the example uses
Twisted
as the project name, since the officialTwisted
library now uses asrc
layout, which contradicts the first "Don't" recommendation: "put your source in a directory called src or lib. This makes it hard to run without installing." This is the whole point (see Ionel Cristian Mărieș's article). -
Maggyero over 4 yearsWhat a great answer, especially the part about non-Python data, thanks!
-
St.Antario over 4 years@pjz Could you please expand your idea? Are you talking about putting
Makefile
at the same level assetup.py
? So if I understands you correctlymake env
automates creatating a newvenv
and install the packages into it... ? -
Notre over 4 years@St.Antario exactly. As mentioned I generally also have a 'test' target to run the tests, and sometimes a 'release' target that looks at the current tag and builds a wheel and sends it to pypi.
-
Martin Thoma about 4 years"
/scripts
or/bin
": I would not do that. You can defineentry_points
in setup.py and have acli.py
or something similar with that as part of the package. -
qrtLs about 4 years+ trend towards
root/src/*
structure: github.com/pypa/sampleproject/commit/… -
qrtLs about 4 yearsfor project structure recs also see setuptools.readthedocs.io/_/downloads/en/latest/pdf
-
Mayou36 about 4 yearsThis post seems rather outdated, I have not found a single project that uses this structure. Could you point me to some? Or maybe add a header in the answer saying it is old?
-
Gustav Rasmussen almost 4 yearsDo: "put your source in a directory called src or lib."
-
SwimBikeRun over 3 yearsActually "This makes it hard to run without installing." is a huge advantage. Everyone should be installing with
pip install -e
, then there's no ambiguity and chance of testing not what you think you're testing -
dawid about 3 yearsThe "don't" section sounds extremely pythonic, however, like many (lazy) Python community practices, are prone to mistakes and incidental complexty - as some commenters pointed out as well.
-
user8491363 about 3 yearsIn this structure, how does any file inside
/code/sandman/sandman/
import something in/code/sandman/docs/
? Say, I want to importconfig.py
fromsandman.py
. How should I do that? -
carloswm85 almost 3 yearsIs the
python_boilerplate_template
still a valid solution in 2021? How long can such solution be still valid. Probably my question is a little vague, but still interesting to know for those like me, who are just students or junior developers. Also, I'm a little worried that the las update that repo had was around 2014 (around 7 years ago at the time of writing). -
KT. almost 3 years@carloswm85 It is for sure still a valid solution, however not without its flair of "old-schoolness", as Buildout has considerably lost in popularity (together with Django, its most famous adopter). Pipenv has been a widely popularized tool to fill the same niche, but its adoption also seems to vane nowadays - many projects resort to manually managing barebones virtualenv + requirements.txt files. It is not that complicated after all, and both Pipenv and Buildout are quite thin opinionated wrappers around this workflow. Project.toml is another "modern" thing that is missing from the template.
-
Yllier123 over 2 yearsThe link provided to the blog post is broken, here it is on the WayBackMachine with comments
-
Franz Kurt over 2 yearsThe link is not available
-
mzjn over 2 yearsThe link to as.ynchrono.us/2007/12/… is dead.