Use Sphinx autosummary recursively to generate API documentation

Question

I want to use Sphinx's autosummary extension and templates to generate API docs recursively from docstrings. I want separate pages for each module, class, method, property and function. But it doesn't detect my templates at all. In fact, if I just remove the module.rst file from _templates/autosummary/, it renders the whole file exactly the same way as before. I've followed this SO question to the letter. If you're interested, the full repository is on GitHub.

Edit: It seems it does generate a different file, I had to delete docs/_autosummary for it to read the new template. However, now it generates a file with the sparse header and description header. It doesn't go into the {% if classes %} and {% if functions %} directives.

My directory structure is as follows:

• sparse
• docs
  • conf.py
  • index.rst
  • modules.rst
  • _templates/autosummary/module.rst

Here are the relevant files so far:

index.rst:

.. sparse documentation master file, created by
   sphinx-quickstart on Fri Dec 29 20:58:03 2017.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Welcome to sparse's documentation!
==================================

.. toctree::
   :maxdepth: 2
   :caption: Contents:

   modules

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

modules.rst:

API Reference
=============

Modules
-------

.. autosummary::
   :toctree: _autosummary

   sparse

_templates/autosummary/module.rst:

{{ fullname | escape | underline }}

Description
-----------

.. automodule:: {{ fullname | escape }}

{% if classes %}
Classes
-------
.. autosummary:
    :toctree: _autosummary

    {% for class in classes %}
        {{ class }}
    {% endfor %}

{% endif %}

{% if functions %}
Functions
---------
.. autosummary:
    :toctree: _autosummary

    {% for function in functions %}
        {{ function }}
    {% endfor %}

{% endif %}

Solution 1

From Sphinx version 3.1 (June 2020), you can use the new :recursive: option to get sphinx.ext.autosummary to automatically detect every module in your package, however deeply nested, and automatically generate documentation for every attribute, class, function and exception in that module.

See my answer here: https://stackoverflow.com/a/62613202/12014259

Solution 2

I ended up needing the following files:

modules.rst:

API Reference
=============

.. rubric:: Modules

.. autosummary::
   :toctree: generated

   sparse

_templates/autosummary/module.rst:

{{ fullname | escape | underline }}

.. rubric:: Description

.. automodule:: {{ fullname }}

.. currentmodule:: {{ fullname }}

{% if classes %}
.. rubric:: Classes

.. autosummary::
    :toctree: .
    {% for class in classes %}
    {{ class }}
    {% endfor %}

{% endif %}

{% if functions %}
.. rubric:: Functions

.. autosummary::
    :toctree: .
    {% for function in functions %}
    {{ function }}
    {% endfor %}

{% endif %}

_templates/autosummary/class.rst:

{{ fullname | escape | underline}}

.. currentmodule:: {{ module }}

.. autoclass:: {{ objname }}

   {% block methods %}
   {% block attributes %}
   {% if attributes %}
   .. HACK -- the point here is that we don't want this to appear in the output, but the autosummary should still generate the pages.
      .. autosummary::
         :toctree:
      {% for item in all_attributes %}
         {%- if not item.startswith('_') %}
         {{ name }}.{{ item }}
         {%- endif -%}
      {%- endfor %}
   {% endif %}
   {% endblock %}

   {% if methods %}
   .. HACK -- the point here is that we don't want this to appear in the output, but the autosummary should still generate the pages.
      .. autosummary::
         :toctree:
      {% for item in all_methods %}
         {%- if not item.startswith('_') or item in ['__call__'] %}
         {{ name }}.{{ item }}
         {%- endif -%}
      {%- endfor %}
   {% endif %}
   {% endblock %}

_templates/autosummary/base.rst:

{{ fullname | escape | underline}}

.. currentmodule:: {{ module }}

.. auto{{ objtype }}:: {{ objname }}

I also needed to go to sphinx/ext/autosummary/generate.py and set imported_members=True in the function generate_autosummary_docs.

If you're not using numpydoc like me, you might need to remove the .. HACK directives.

Comments

[Steve Piercy]

Why don't you use sphinx-apidoc?

[Hameer Abbasi]

If sphinx-apidoc can do something similar to what I want (with templates and member summaries), then I shall. Unfortunately I'm not too good with Sphinx and an example would be useful. But as I understand it, sphinx-apidoc is equivalent to using autosummary with autosummary_generate = True.

[Steve Piercy]

What have you tried? Have you read the docs for sphinx-apidoc to see if it meets your needs?

[Hameer Abbasi]

@StevePiercy With quite a bit of fiddling, I got it to work. See my answer below.

[mzjn]

I cloned your GitHub project and it worked for me too. Setting imported_members=True in the generate_autosummary_docs function seems to be the key. But it is problematic that it should be necessary to edit the Sphinx source code to make it work.

[mzjn]

I just saw github.com/sphinx-doc/sphinx/issues/4372. You have __all__ = ["COO", "tensordot", "concatenate", "stack", "dot", "triu", "tril"] in __init__.py and that should be enough IMHO. As far as I can tell, autosummary does not take __all__ into account.

[Hameer Abbasi]

Yes, it should either take __all__ into account or provide a way to include imported members.

[mar10]

Great answer, very helpful! Are you aware of a way to add the documented module attributes (like constant definitions) to the output in modules.rst?

[Hameer Abbasi]

@mar10 You would add an attributes section similar to the functions and classes section in _templates/autosummary/module.rst. I haven't tried it myself, but I'm pretty sure it will work.

[mar10]

@HameerAbbasi attributes seems to be empty, but members has content. I can filter out like if item not in all_functions and item not in all_classes ... and get mostly what I want (except for imported module names and symbols, which I did not figure out how to exclude)