How should I organize Python source code?

Question

I'm getting started with Python (it's high time I give it a shot), and I'm looking for some best practices.

My first project is a queue which runs command-line experiments in multiple threads. I'm starting to get a very long main.py file, and I'd like to break it up. In general, I'm looking for: How do python programmers organize multiple source files? Is there a particular structure that works for you?

My specific questions include:

1. Should each class be in a separate file?
2. How should I organize unit tests relative to source code?
3. Where should I put doc comments, specifically those for command-line operation?
4. If I use multiple directories, how do I import classes between them?

I can probably draw some of my own conclusions here by trial and error, but I'd rather start from something good.

Solution 1

The article Eric pointed to is awesome because it covers details of organising large Python code bases.

If you've landed here from Google and are trying to find out how to split one large source file into multiple, more manageable, files I'll summarise the process briefly.

Assume you currently have everything in a file called main.py:

• Create another source file in the same folder (let's call ours utils.py for this example)
• Move whatever classes, functions, statements, etc you need from main.py into utils.py
• In main.py add a single line at the top: import utils

Conceptually what this does is to create a new module called utils in another source file. You can then import it wherever it's needed.

Solution 2

The way you should organise your code and tests is exactly the same you would for any OO language.

Answers from the way I do it. It may not be right but works for me

1. Depends on how your functionality is split. For my main python app I have 1 file with classes for the entry points and then packages of different bits of functionality
2. I use PyDev for eclipse and organise it like I would for Java.

>  Workspace
>     |
>     |-Src
>     |   |-Package1
>     |   |-Package2
>     |   |-main.py
>     |-Test
>         |-TestPackage1
>         |-TestPackage2

3. Use DocString everywhere to keep track of everything
4. After making sure that the relevant __init__.py files are in the folders. its just a simple case of from module import class

Comments

[Nikola Smiljanić]

This will explain a couple of things about organizing your code docs.python.org/tutorial/modules.html

[rda3mon]

Here is some more useful info from python docs. <br> docs.python.org/3/tutorial/modules.html#packages

[Andres Jaan Tack]

This question is in search of a broadly-accepted convention specifically in the Python community. The answer is not a matter of opinion, though like most answers it could change with time. I suggest this be re-opened or at the very least the original answer undeleted.

[Roboprog]

One caveat, though: java takes kind of a dictatorial relationship with packages, files and classes. Sometimes I end up with way more source files than I would really want. The conventions of some organizations -- e.g. - avoid (nested) inner classes or "helper" classes lower in the file -- make this worse, beyond the compiler's requirements. Keep it orderly, and a hierarchy is useful, but try to avoid make-work.

[Ssam]

Do you happen to remember the article Eric pointed to? I can't seem to find an Eric on this question/answer

[smci]

@DrewNoakes: I think it was deleted for being a link-only answer; if only he had summarized the article's main points.

[Igor Brejc]

Unfortunately the article is a dead link now :-(. The latest archived version is here: web.archive.org/web/20190714164001/http://…