Sphinx autodoc is not automatic enough

Question

I'm trying to use Sphinx to document a 5,000+ line project in Python. It has about 7 base modules. As far as I know, In order to use autodoc I need to write code like this for each file in my project:

.. automodule:: mods.set.tests
    :members:
    :show-inheritance:

This is way too tedious because I have many files. It would be much easier if I could just specify that I wanted the 'mods' package to be documented. Sphinx could then recursively go through the package and make a page for each submodule.

Is there a feature like this? If not I could write a script to make all the .rst files, but that would take up a lot of time.

Solution 1

You can check this script that I've made. I think it can help you.

This script parses a directory tree looking for python modules and packages and creates ReST files appropriately to create code documentation with Sphinx. It also creates a modules index.

UPDATE

This script is now part of Sphinx 1.1 as apidoc.

Solution 2

From Sphinx version 3.1 (June 2020), sphinx.ext.autosummary (finally!) has automatic recursion.

So no need to hard code module names or rely on 3rd party libraries like Sphinx AutoAPI or Sphinx AutoPackageSummary for their automatic package detection any more.

Example Python 3.7 package to document (see code on Github and result on ReadTheDocs):

mytoolbox
|-- mypackage
|   |-- __init__.py
|   |-- foo.py
|   |-- mysubpackage
|       |-- __init__.py
|       |-- bar.py
|-- doc
|   |-- source
|       |--index.rst
|       |--conf.py
|       |-- _templates
|           |-- custom-module-template.rst
|           |-- custom-class-template.rst

conf.py:

import os
import sys
sys.path.insert(0, os.path.abspath('../..'))  # Source code dir relative to this file

extensions = [
    'sphinx.ext.autodoc',  # Core library for html generation from docstrings
    'sphinx.ext.autosummary',  # Create neat summary tables
]
autosummary_generate = True  # Turn on sphinx.ext.autosummary

# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']

index.rst (note new :recursive: option):

Welcome to My Toolbox
=====================

Some words.

.. autosummary::
   :toctree: _autosummary
   :template: custom-module-template.rst
   :recursive:

   mypackage

This is sufficient to automatically summarise every module in the package, however deeply nested. For each module, it then summarises every attribute, function, class and exception in that module.

Oddly, though, the default sphinx.ext.autosummary templates don't go on to generate separate documentation pages for each attribute, function, class and exception, and link to them from the summary tables. It's possible to extend the templates to do this, as shown below, but I can't understand why this isn't the default behaviour - surely that's what most people would want..? I've raised it as a feature request.

I had to copy the default templates locally, and then add to them:

• Copy site-packages/sphinx/ext/autosummary/templates/autosummary/module.rst to mytoolbox/doc/source/_templates/custom-module-template.rst
• Copy site-packages/sphinx/ext/autosummary/templates/autosummary/class.rst to mytoolbox/doc/source/_templates/custom-class-template.rst

The hook into custom-module-template.rst is in index.rst above, using the :template: option. (Delete that line to see what happens using the default site-packages templates.)

custom-module-template.rst (additional lines noted on the right):

{{ fullname | escape | underline}}

.. automodule:: {{ fullname }}
  
   {% block attributes %}
   {% if attributes %}
   .. rubric:: Module Attributes

   .. autosummary::
      :toctree:                                          <-- add this line
   {% for item in attributes %}
      {{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

   {% block functions %}
   {% if functions %}
   .. rubric:: {{ _('Functions') }}

   .. autosummary::
      :toctree:                                          <-- add this line
   {% for item in functions %}
      {{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

   {% block classes %}
   {% if classes %}
   .. rubric:: {{ _('Classes') }}

   .. autosummary::
      :toctree:                                          <-- add this line
      :template: custom-class-template.rst               <-- add this line
   {% for item in classes %}
      {{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

   {% block exceptions %}
   {% if exceptions %}
   .. rubric:: {{ _('Exceptions') }}

   .. autosummary::
      :toctree:                                          <-- add this line
   {% for item in exceptions %}
      {{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

{% block modules %}
{% if modules %}
.. rubric:: Modules

.. autosummary::
   :toctree:
   :template: custom-module-template.rst                 <-- add this line
   :recursive:
{% for item in modules %}
   {{ item }}
{%- endfor %}
{% endif %}
{% endblock %}

custom-class-template.rst (additional lines noted on the right):

{{ fullname | escape | underline}}

.. currentmodule:: {{ module }}

.. autoclass:: {{ objname }}
   :members:                                    <-- add at least this line
   :show-inheritance:                           <-- plus I want to show inheritance...
   :inherited-members:                          <-- ...and inherited members too

   {% block methods %}
   .. automethod:: __init__

   {% if methods %}
   .. rubric:: {{ _('Methods') }}

   .. autosummary::
   {% for item in methods %}
      ~{{ name }}.{{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

   {% block attributes %}
   {% if attributes %}
   .. rubric:: {{ _('Attributes') }}

   .. autosummary::
   {% for item in attributes %}
      ~{{ name }}.{{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

Solution 3

I do not know whether Sphinx had had autosummary extension at the time original question was asked, but for now it is quite possible to set up automatic generation of that kind without using sphinx-apidoc or similar script. Below there are settings which work for one of my projects.

1. Enable autosummary extension (as well as autodoc) in conf.py file and set its autosummary_generate option to True. This may be enough if you're not using custom *.rst templates. Otherwise add your templates directory to exclude list, or autosummary will try to treat them as input files (which seems to be a bug).

   extensions = ['sphinx.ext.autodoc', 'sphinx.ext.autosummary']
   autosummary_generate = True
   templates_path = [ '_templates' ]
   exclude_patterns = ['_build', '_templates']
   

2. Use autosummary:: in TOC tree in your index.rst file. In this example documentation for modules project.module1 and project.module2 will be generated automatically and placed into _autosummary directory.

   PROJECT
   =======
   
   .. toctree::
   
   .. autosummary::
      :toctree: _autosummary
   
      project.module1
      project.module2
   

3. By default autosummary will generate only very short summaries for modules and their functions. To change that you can put a custom template file into _templates/autosummary/module.rst (which will be parsed with Jinja2):

   {{ fullname }}
   {{ underline }}
   
   .. automodule:: {{ fullname }}
       :members:
   

In conclusion, there is no need to keep _autosummary directory under version control. Also, you may name it anything you want and place it anywhere in the source tree (putting it below _build will not work, though).

Solution 4

Sphinx AutoAPI does exactly this.

Solution 5

In each package, the __init__.py file can have .. automodule:: package.module components for each part of the package.

Then you can .. automodule:: package and it mostly does what you want.

View more solutions

Comments

[Sakie]

No one said it was hard. OP said it was tedious, which it is. Given that other doc systems can do this, it's not unreasonable.

[K3---rnc]

Just use pdoc.

[Cory Walker]

do I just put that string in triple quotes in init.py?

[user1066101]

@Cory Walker: It's not "a" string. You can -- and should -- be putting triple-quoted docstrings in every single file. Every one. That includes the __init__.py files in your packages. The docstring can include ANY Sphinx documentation directives, including .. automodule:: for modules within the package.

[Cerin]

Where are you supposed to output the files to? I tried outputting them to Sphinx's default _build folder, but running sphinx-build -b html . ./_build doesn't pick them up.

[Etienne]

You should put them in the source directory (. in your case). The _build directory is where the HTML files will be created. Check for more info: sphinx.pocoo.org/tutorial.html#running-the-build

[jbenet]

@Erienne: fantastic script! just what i was looking for. Wish it generated headings for individual classes (the regular sphinx look isn't nice to classes. they get lost in larger modules)

[slacy]

Even sphinx-apidoc is pretty rudimentary. For a package with one or two modules, it works okay, but we've got modules nested deeply, and sphinx-apidoc produces some pretty unmanageable output.

[Etienne]

@slacy It's true that sphinx-apidoc is relatively rudimentary but I'm curious to know more precisely what you mean by "unmanageable output"? I use it on a not so small code base without any problems (but modules were not nested too deeply).

[Ciro Santilli OurBigBook.com]

how to use the shpinx-apidoc output? I've got mod.rst and modules.rst on the same dir as conf.py and index.rst. You could put/ this info on the manual page @Etienne.

[Ciro Santilli OurBigBook.com]

self answering: add .. include:: modules.rst to your index.rst

[Ciro Santilli OurBigBook.com]

self answering again: if you are on a new project, use apidoc -F -o desired-doc-location py-source-root, and this generates the the doc templates with a working index.rst. The generated Makefile however does not regenerate module list as of current version (if you add a new file, you need to rerun sphinx-apidoc)

[Brown]

This was a huge help. In point 2, where you have "project.module1" and "project.module2", is there a way to automagically generate that list for every module in a given package? To just put "project" and have it sniff out "module1" and "module2"?

[Alisdair Robertson]

Pretty surprised I can't find an answer to this anywhere, dod you ever work it out @Brown?

[Brown]

@AlisdairRobertson No, but the autosummary solution provided ended up being more than adequate for my needs. The only other thing I thought of doing was to write a script to generate the index.rst file and autodetect the module names. However, in practice, the list of modules doesn't change that often, so just editing one file once in a while isn't that unreasonable. I'm sure I already spent much more time looking for a solution than it takes to just edit that one file!

[ropeladder]

Oh my goodness! This works so much better than anything else. Note that this is NOT "autodoc" or "apidoc", it's a completely different extension.

[nealmcb]

Ditto. This puts the "auto" in "autodoc".... Here is all our project had to do to switch: Switch to autoapi from autodoc by nealmcb · Pull Request \#7 · gwexploratoryaudits/r2b2

[samlaf]

Thank you so much for this!!! I was almost about to give up and then finally found this. One question. I was using sys.path.insert(0, os.path.abspath('../../mypackage')) but you only use '../..'. Why does it not make a difference?

[James Leedham]

No problem, glad it helped. Sphinx nearly drove me insane until I got it figured; now it all works beautifully. In index.rst, you need to explicitly name the package where you want :recursive: docstring extraction to start, so in my example mypackage. In conf.py, you just provide the route to that location. That's because in my example mypackage is a top-level folder in mytoolbox, but if it was (eg.) nested in a source/mypackage subfolder then conf.py would read ../../source.

[SKG]

@jamleed This has been very useful!. Thanks for taking the time and writing Sphinx AutoPackageSummary. You saved me a lot of time. In that github project, the make html part (make.bat) was not available, so "make html" command did not work out of the box. You might have to add some instructions around that or upload that file as well. Otherwise, it worked as a charm. (Edited - included the missing file name)

[James Leedham]

Ok, thanks for pointing that out. A function of my not thinking about Windows users - sorry! Glad the project was of use...

[rickstaa]

This answer is gold, thanks a lot! Hope your feature request gets implemented. It should indeed be the default behaviour.

[FullmetalEngineer]

Just a small improvement to custom-module-template.rst, if you add :members: under the .. automodule:: {{ fullname }} (line 3), the documentation for that module will automatically be populated with members (classes, functions, etc.) in addition to the toctree. This gives users the option to quickly scan through the documentation in addition to pinpointing functions using the toctree.

[Hyperplane]

How do I prevent sphinx from documenting 3rd party classes when using this in conjunction with autosummary_imported_members = True? I want this option to get documentation for imports in the __init__.py files. I already added __all__ lists to all my modules but it still creates .rst's for 3rd party classes

[Vineet]

It didn't work for code structure given here; couldn't open docs for functions and classes in codebase

[James Leedham]

@Vineet what's in your __init__.py files? If lots of import statements, then I've seen Sphinx fail to follow them properly, as per this bug: github.com/sphinx-doc/sphinx/issues/9069 (though I'm unsure what the exact problem is...) A

[James Leedham]

Also, what happens if you remove all the transform module code, and the transform statement in index.rst, and just run make html on the extract module?

[Vineet]

@JamesLeedham my __init__.pys are empty. On removing transform module, similar behaviour was observed as one can see in autosummary - I am unable to view documentation of functions and classes on clicking on them. This does not happen when I specify individual files like - extract.folder_1.a.

[Karim Akhnoukh]

Thank you so much for you detailed answer. It really helped a lot :)

[lexalenka]

The script link is broken.

[lexalenka]

Are there no other steps to this? Updating to this is still only doing current directory and not recursive. I also see: WARNING: Error in "autosummary" directive: unknown option: "recursive". Current directory html is made only.

[Etienne]

Yes, as mention in the UPDATE, the script is now part of Sphinx as apidoc. I strikethrough the line to be more clear.

[dheinz]

Thank you so much for this answer. This literally fully automates the documentation :) Incredible what one recursive feature is able to do :)

[endolith]

Links are not answers, please elaborate

[Caio Castro]

Amazing answer. Worked right out the gate, thanks!

[Phil]

What commandline-scripts do you run to get this working?

[James Leedham]

@Phil, if you pull the repo and follow these instructions to build it locally, you should be able to unpick all the commands: github.com/JamesALeedham/Sphinx-Autosummary-Recursion/tree/…

[Diego F Medina]

Special attention for the "Copy (...)/module.rst" and "class.rst" parts, as the files have changed recently and copying the whole code above might not work correctly.