prefetch_related for multiple Levels

Question

If my Models look like:

class Publisher(models.Model):
    pass

class Book(models.Model):
    publisher = models.ForeignKey(Publisher)

class Page(models.Model):
    book = models.ForeignKey(Book)

and I would like to get the queryset for Publisher I do Publisher.object.all(). If then want to make sure to prefetch I can do:

Publisher.objects.all().prefetch_related('book_set')`

My questions are:

1. Is there a way to do this prefetching using select_related or must I use prefetch_related?
2. Is there a way to prefetch the page_set? This does not work:

Publisher.objects.all().prefetch_related('book_set', 'book_set_page_set')

Solution 1

1. No, you cannot use select_related for a reverse relation. select_related does a SQL join, so a single record in the main queryset needs to reference exactly one in the related table (ForeignKey or OneToOne fields). prefetch_related actually does a totally separate second query, caches the results, then "joins" it into the queryset in python. So it is needed for ManyToMany or reverse ForeignKey fields.

2. Have you tried two underscores to do the multi level prefetches? Like this: Publisher.objects.all().prefetch_related('book_set', 'book_set__page_set')

Solution 2

Since Django 1.7, instances of django.db.models.Prefetch class can be used as an argument of .prefetch_related. Prefetch object constructor has a queryset argument that allows to specify nested multiple levels prefetches like that:

Project.objects.filter(
        is_main_section=True
    ).select_related(
        'project_group'
    ).prefetch_related(
        Prefetch(
            'project_group__project_set',
            queryset=Project.objects.prefetch_related(
                Prefetch(
                    'projectmember_set',
                    to_attr='projectmember_list'
                )
            ),
            to_attr='project_list'
        )
    )

It is stored into attributes with _list suffix because I use ListQuerySet to process prefetch results (filter / order).

Comments

[Alex Rothberg]

1. If Page has a OneToOne field to TextContent, would it be: ...prefetch_related('book_set__page_set__text_contents') or ...select_related('book_set__page_set__text_contents')

[jproffitt]

I believe it would be the second version.

[Mohammed Shareef C]

This is a really brilliant answer and lets you save a lot on db traffic. I don't understand why this answer didn't get much attention

[Dmitriy Sintsov]

See github.com/Dmitri-Sintsov/django-jinja-knockout/…

[Manish Shah]

Hi @DmitriySintsov Can I please get your help here:- stackoverflow.com/questions/68682472/…

[dKen]

-1. The solution here uses unknown models and an unknown structure. It would be much easier to understand if you used the models that OP provided, or explain the structure of your project example. Very happy to undo the -1 if we get that clarity.