Separation of business logic and data access in django

Question

I am writing a project in Django and I see that 80% of the code is in the file models.py. This code is confusing and, after a certain time, I cease to understand what is really happening.

Here is what bothers me:

1. I find it ugly that my model level (which was supposed to be responsible only for the work with data from a database) is also sending email, walking on API to other services, etc.
2. Also, I find it unacceptable to place business logic in the view, because this way it becomes difficult to control. For example, in my application there are at least three ways to create new instances of User, but technically it should create them uniformly.
3. I do not always notice when the methods and properties of my models become non-deterministic and when they develop side effects.

Here is a simple example. At first, the User model was like this:

class User(db.Models):

    def get_present_name(self):
        return self.name or 'Anonymous'

    def activate(self):
        self.status = 'activated'
        self.save()

Over time, it turned into this:

class User(db.Models):

    def get_present_name(self): 
        # property became non-deterministic in terms of database
        # data is taken from another service by api
        return remote_api.request_user_name(self.uid) or 'Anonymous' 

    def activate(self):
        # method now has a side effect (send message to user)
        self.status = 'activated'
        self.save()
        send_mail('Your account is activated!', '…', [self.email])

What I want is to separate entities in my code:

1. Entities of my database, persistence level: What data does my application keep?
2. Entities of my application, business logic level: What does my application do?

What are the good practices to implement such an approach that can be applied in Django?

Solution 1

It seems like you are asking about the difference between the data model and the domain model – the latter is where you can find the business logic and entities as perceived by your end user, the former is where you actually store your data.

Furthermore, I've interpreted the 3rd part of your question as: how to notice failure to keep these models separate.

These are two very different concepts and it's always hard to keep them separate. However, there are some common patterns and tools that can be used for this purpose.

About the Domain Model

The first thing you need to recognize is that your domain model is not really about data; it is about actions and questions such as "activate this user", "deactivate this user", "which users are currently activated?", and "what is this user's name?". In classical terms: it's about queries and commands.

Thinking in Commands

Let's start by looking at the commands in your example: "activate this user" and "deactivate this user". The nice thing about commands is that they can easily be expressed by small given-when-then scenario's:

given an inactive user
when the admin activates this user
then the user becomes active
and a confirmation e-mail is sent to the user
and an entry is added to the system log
(etc. etc.)

Such scenario's are useful to see how different parts of your infrastructure can be affected by a single command – in this case your database (some kind of 'active' flag), your mail server, your system log, etc.

Such scenario's also really help you in setting up a Test Driven Development environment.

And finally, thinking in commands really helps you create a task-oriented application. Your users will appreciate this :-)

Expressing Commands

Django provides two easy ways of expressing commands; they are both valid options and it is not unusual to mix the two approaches.

The service layer

The service module has already been described by @Hedde. Here you define a separate module and each command is represented as a function.

services.py

def activate_user(user_id):
    user = User.objects.get(pk=user_id)

    # set active flag
    user.active = True
    user.save()

    # mail user
    send_mail(...)

    # etc etc

Using forms

The other way is to use a Django Form for each command. I prefer this approach, because it combines multiple closely related aspects:

• execution of the command (what does it do?)
• validation of the command parameters (can it do this?)
• presentation of the command (how can I do this?)

forms.py

class ActivateUserForm(forms.Form):

    user_id = IntegerField(widget = UsernameSelectWidget, verbose_name="Select a user to activate")
    # the username select widget is not a standard Django widget, I just made it up

    def clean_user_id(self):
        user_id = self.cleaned_data['user_id']
        if User.objects.get(pk=user_id).active:
            raise ValidationError("This user cannot be activated")
        # you can also check authorizations etc. 
        return user_id

    def execute(self):
        """
        This is not a standard method in the forms API; it is intended to replace the 
        'extract-data-from-form-in-view-and-do-stuff' pattern by a more testable pattern. 
        """
        user_id = self.cleaned_data['user_id']

        user = User.objects.get(pk=user_id)

        # set active flag
        user.active = True
        user.save()

        # mail user
        send_mail(...)

        # etc etc

Thinking in Queries

You example did not contain any queries, so I took the liberty of making up a few useful queries. I prefer to use the term "question", but queries is the classical terminology. Interesting queries are: "What is the name of this user?", "Can this user log in?", "Show me a list of deactivated users", and "What is the geographical distribution of deactivated users?"

Before embarking on answering these queries, you should always ask yourself this question, is this:

• a presentational query just for my templates, and/or
• a business logic query tied to executing my commands, and/or
• a reporting query.

Presentational queries are merely made to improve the user interface. The answers to business logic queries directly affect the execution of your commands. Reporting queries are merely for analytical purposes and have looser time constraints. These categories are not mutually exclusive.

The other question is: "do I have complete control over the answers?" For example, when querying the user's name (in this context) we do not have any control over the outcome, because we rely on an external API.

Making Queries

The most basic query in Django is the use of the Manager object:

User.objects.filter(active=True)

Of course, this only works if the data is actually represented in your data model. This is not always the case. In those cases, you can consider the options below.

Custom tags and filters

The first alternative is useful for queries that are merely presentational: custom tags and template filters.

template.html

<h1>Welcome, {{ user|friendly_name }}</h1>

template_tags.py

@register.filter
def friendly_name(user):
    return remote_api.get_cached_name(user.id)

Query methods

If your query is not merely presentational, you could add queries to your services.py (if you are using that), or introduce a queries.py module:

queries.py

def inactive_users():
    return User.objects.filter(active=False)


def users_called_publysher():
    for user in User.objects.all():
        if remote_api.get_cached_name(user.id) == "publysher":
            yield user 

Proxy models

Proxy models are very useful in the context of business logic and reporting. You basically define an enhanced subset of your model. You can override a Manager’s base QuerySet by overriding the Manager.get_queryset() method.

models.py

class InactiveUserManager(models.Manager):
    def get_queryset(self):
        query_set = super(InactiveUserManager, self).get_queryset()
        return query_set.filter(active=False)

class InactiveUser(User):
    """
    >>> for user in InactiveUser.objects.all():
    …        assert user.active is False 
    """

    objects = InactiveUserManager()
    class Meta:
        proxy = True

Query models

For queries that are inherently complex, but are executed quite often, there is the possibility of query models. A query model is a form of denormalization where relevant data for a single query is stored in a separate model. The trick of course is to keep the denormalized model in sync with the primary model. Query models can only be used if changes are entirely under your control.

models.py

class InactiveUserDistribution(models.Model):
    country = CharField(max_length=200)
    inactive_user_count = IntegerField(default=0)

The first option is to update these models in your commands. This is very useful if these models are only changed by one or two commands.

forms.py

class ActivateUserForm(forms.Form):
    # see above
   
    def execute(self):
        # see above
        query_model = InactiveUserDistribution.objects.get_or_create(country=user.country)
        query_model.inactive_user_count -= 1
        query_model.save()

A better option would be to use custom signals. These signals are of course emitted by your commands. Signals have the advantage that you can keep multiple query models in sync with your original model. Furthermore, signal processing can be offloaded to background tasks, using Celery or similar frameworks.

signals.py

user_activated = Signal(providing_args = ['user'])
user_deactivated = Signal(providing_args = ['user'])

forms.py

class ActivateUserForm(forms.Form):
    # see above
   
    def execute(self):
        # see above
        user_activated.send_robust(sender=self, user=user)

models.py

class InactiveUserDistribution(models.Model):
    # see above

@receiver(user_activated)
def on_user_activated(sender, **kwargs):
        user = kwargs['user']
        query_model = InactiveUserDistribution.objects.get_or_create(country=user.country)
        query_model.inactive_user_count -= 1
        query_model.save()
    

Keeping it clean

When using this approach, it becomes ridiculously easy to determine if your code stays clean. Just follow these guidelines:

• Does my model contain methods that do more than managing database state? You should extract a command.
• Does my model contain properties that do not map to database fields? You should extract a query.
• Does my model reference infrastructure that is not my database (such as mail)? You should extract a command.

The same goes for views (because views often suffer from the same problem).

• Does my view actively manage database models? You should extract a command.

Some References

Django documentation: proxy models

Django documentation: signals

Architecture: Domain Driven Design

Solution 2

I usually implement a service layer in between views and models. This acts like your project's API and gives you a good helicopter view of what is going on. I inherited this practice from a colleague of mine that uses this layering technique a lot with Java projects (JSF), e.g:

models.py

class Book:
   author = models.ForeignKey(User)
   title = models.CharField(max_length=125)

   class Meta:
       app_label = "library"

services.py

from library.models import Book

def get_books(limit=None, **filters):
    """ simple service function for retrieving books can be widely extended """
    return Book.objects.filter(**filters)[:limit]  # list[:None] will return the entire list

views.py

from library.services import get_books

class BookListView(ListView):
    """ simple view, e.g. implement a _build and _apply filters function """
    queryset = get_books()

Mind you, I usually take models, views and services to module level and separate even further depending on the project's size

Solution 3

First of all, Don't repeat yourself.

Then, please be careful not to overengineer, sometimes it is just a waste of time, and makes someone lose focus on what is important. Review the zen of python from time to time.

Take a look at active projects

• more people = more need to organize properly
• the django repository they have a straightforward structure.
• the pip repository they have a straigtforward directory structure.

• the fabric repository is also a good one to look at.

  • you can place all your models under yourapp/models/logicalgroup.py
• e.g User, Group and related models can go under yourapp/models/users.py
• e.g Poll, Question, Answer ... could go under yourapp/models/polls.py
• load what you need in __all__ inside of yourapp/models/__init__.py

More about MVC

• model is your data
  • this includes your actual data
  • this also includes your session / cookie / cache / fs / index data
• user interacts with controller to manipulate the model
  • this could be an API, or a view that saves/updates your data
  • this can be tuned with request.GET / request.POST ...etc
  • think paging or filtering too.
• the data updates the view
  • the templates take the data and format it accordingly
  • APIs even w/o templates are part of the view; e.g. tastypie or piston
  • this should also account for the middleware.

Take advantage of middleware / templatetags

• If you need some work to be done for each request, middleware is one way to go.
  • e.g. adding timestamps
  • e.g. updating metrics about page hits
  • e.g. populating a cache
• If you have snippets of code that always reoccur for formatting objects, templatetags are good.
  • e.g. active tab / url breadcrumbs

Take advantage of model managers

• creating User can go in a UserManager(models.Manager).
• gory details for instances should go on the models.Model.
• gory details for queryset could go in a models.Manager.
• you might want to create a User one at a time, so you may think that it should live on the model itself, but when creating the object, you probably don't have all the details:

Example:

class UserManager(models.Manager):
   def create_user(self, username, ...):
      # plain create
   def create_superuser(self, username, ...):
      # may set is_superuser field.
   def activate(self, username):
      # may use save() and send_mail()
   def activate_in_bulk(self, queryset):
      # may use queryset.update() instead of save()
      # may use send_mass_mail() instead of send_mail()

Make use of forms where possible

A lot of boilerplate code can be eliminated if you have forms that map to a model. The ModelForm documentation is pretty good. Separating code for forms from model code can be good if you have a lot of customization (or sometimes avoid cyclic import errors for more advanced uses).

Use management commands when possible

• e.g. yourapp/management/commands/createsuperuser.py
• e.g. yourapp/management/commands/activateinbulk.py

if you have business logic, you can separate it out

• django.contrib.auth uses backends, just like db has a backend...etc.
• add a setting for your business logic (e.g. AUTHENTICATION_BACKENDS)
• you could use django.contrib.auth.backends.RemoteUserBackend
• you could use yourapp.backends.remote_api.RemoteUserBackend
• you could use yourapp.backends.memcached.RemoteUserBackend
• delegate the difficult business logic to the backend
• make sure to set the expectation right on the input/output.
• changing business logic is as simple as changing a setting :)

backend example:

class User(db.Models):
    def get_present_name(self): 
        # property became not deterministic in terms of database
        # data is taken from another service by api
        return remote_api.request_user_name(self.uid) or 'Anonymous' 

could become:

class User(db.Models):
   def get_present_name(self):
      for backend in get_backends():
         try:
            return backend.get_present_name(self)
         except: # make pylint happy.
            pass
      return None

more about design patterns

• there's already a good question about design patterns
• a very good video about practical design patterns
• django's backends are obvious use of delegation design pattern.

more about interface boundaries

• Is the code you want to use really part of the models? -> yourapp.models
• Is the code part of business logic? -> yourapp.vendor
• Is the code part of generic tools / libs? -> yourapp.libs
• Is the code part of business logic libs? -> yourapp.libs.vendor or yourapp.vendor.libs
• Here is a good one: can you test your code independently?
  • yes, good :)
  • no, you may have an interface problem
  • when there is clear separation, unittest should be a breeze with the use of mocking
• Is the separation logical?
  • yes, good :)
  • no, you may have trouble testing those logical concepts separately.
• Do you think you will need to refactor when you get 10x more code?
  • yes, no good, no bueno, refactor could be a lot of work
  • no, that's just awesome!

In short, you could have

• yourapp/core/backends.py
• yourapp/core/models/__init__.py
• yourapp/core/models/users.py
• yourapp/core/models/questions.py
• yourapp/core/backends.py
• yourapp/core/forms.py
• yourapp/core/handlers.py
• yourapp/core/management/commands/__init__.py
• yourapp/core/management/commands/closepolls.py
• yourapp/core/management/commands/removeduplicates.py
• yourapp/core/middleware.py
• yourapp/core/signals.py
• yourapp/core/templatetags/__init__.py
• yourapp/core/templatetags/polls_extras.py
• yourapp/core/views/__init__.py
• yourapp/core/views/users.py
• yourapp/core/views/questions.py
• yourapp/core/signals.py
• yourapp/lib/utils.py
• yourapp/lib/textanalysis.py
• yourapp/lib/ratings.py
• yourapp/vendor/backends.py
• yourapp/vendor/morebusinesslogic.py
• yourapp/vendor/handlers.py
• yourapp/vendor/middleware.py
• yourapp/vendor/signals.py
• yourapp/tests/test_polls.py
• yourapp/tests/test_questions.py
• yourapp/tests/test_duplicates.py
• yourapp/tests/test_ratings.py

or anything else that helps you; finding the interfaces you need and the boundaries will help you.

Solution 4

Django employs a slightly modified kind of MVC. There's no concept of a "controller" in Django. The closest proxy is a "view", which tends to cause confusion with MVC converts because in MVC a view is more like Django's "template".

In Django, a "model" is not merely a database abstraction. In some respects, it shares duty with the Django's "view" as the controller of MVC. It holds the entirety of behavior associated with an instance. If that instance needs to interact with an external API as part of it's behavior, then that's still model code. In fact, models aren't required to interact with the database at all, so you could conceivable have models that entirely exist as an interactive layer to an external API. It's a much more free concept of a "model".

Solution 5

In Django, MVC structure is as Chris Pratt said, different from classical MVC model used in other frameworks, I think the main reason for doing this is avoiding a too strict application structure, like happens in others MVC frameworks like CakePHP.

In Django, MVC was implemented in the following way:

View layer is splitted in two. The views should be used only to manage HTTP requests, they are called and respond to them. Views communicate with the rest of your application (forms, modelforms, custom classes, of in simple cases directly with models). To create the interface we use Templates. Templates are string-like to Django, it maps a context into them, and this context was communicated to the view by the application (when view asks).

Model layer gives encapsulation, abstraction, validation, intelligence and makes your data object-oriented (they say someday DBMS will also). This doesn't means that you should make huge models.py files (in fact a very good advice is to split your models in different files, put them into a folder called 'models', make an '__init__.py' file into this folder where you import all your models and finally use the attribute 'app_label' of models.Model class). Model should abstract you from operating with data, it will make your application simpler. You should also, if required, create external classes, like "tools" for your models.You can also use heritage in models, setting the 'abstract' attribute of your model's Meta class to 'True'.

Where is the rest? Well, small web applications generally are a sort of an interface to data, in some small program cases using views to query or insert data would be enough. More common cases will use Forms or ModelForms, which are actually "controllers". This is not other than a practical solution to a common problem, and a very fast one. It's what a website use to do.

If Forms are not enogh for you, then you should create your own classes to do the magic, a very good example of this is admin application: you can read ModelAmin code, this actually works as a controller. There is not a standard structure, I suggest you to examine existing Django apps, it depends on each case. This is what Django developers intended, you can add xml parser class, an API connector class, add Celery for performing tasks, twisted for a reactor-based application, use only the ORM, make a web service, modify the admin application and more... It's your responsability to make good quality code, respect MVC philosophy or not, make it module based and creating your own abstraction layers. It's very flexible.

My advice: read as much code as you can, there are lots of django applications around, but don't take them so seriously. Each case is different, patterns and theory helps, but not always, this is an imprecise cience, django just provide you good tools that you can use to aliviate some pains (like admin interface, web form validation, i18n, observer pattern implementation, all the previously mentioned and others), but good designs come from experienced designers.

PS.: use 'User' class from auth application (from standard django), you can make for example user profiles, or at least read its code, it will be useful for your case.

View more solutions

Comments

[Konstant]

Read about signals

[Rune FS]

well you removed the tag but you could use DCI to accmplish the seperation of what the system does (the functionality) and what the system is (the data/domain model)

[defuz]

You propose to implement all the business logic in signal callbacks? Unfortunately, not all of the my application can be linked to events in the database.

[defuz]

Rune FS, I tried to use the DCI, but it seemed to me that it do not need much for my project: Context, definition of roles as mixin to the objects, etc. There is an easier way of separation "does" and "is"? Could you give a minimal example?

[arie]

I like the general approach though from my understanding your specific example would usually be implemented as a manager.

[Hedde van der Heide]

@arie not necessarily, perhaps a better example, for a webshop services would include things like generating cart sessions, asynchronous tasks like product ratings calculations, creating and sending emails etcetera

[Chris Wesseling]

How is saying that it's good enough for you helpful to my understanding of yhe question?

[TaiwanGrapefruitTea]

Very insightful! One question/issue: these suggestions imply that all the processing is inside a django website. An alternative architecture is to use django as the web front-end to a backend platform layer (see lethain.com/introduction-to-architecting-systems-for-scale/… ). This platform layer allows you to run code outside of the website framework. The data model, domain model, service layer, etc would all reside in the platform layer. And, given that this layer is outside of django, one might use SQL Alchemy as the ORM for the platform layer instead of django.

[Scott Coates]

It's nice to see an answer incorporating DDD into a django-related question. Just because Django employs ActiveRecord for persistence doesn't mean separation of concerns should go out the window. Great answer.

[Maximus S]

Where do I put services.py file, and how do I access its methods?

[Ivan]

If I want to validate that the looged user is the owner of an object before deleting that object, should I check that in the view or in the form/service module?

[publysher]

@Ivan: both. It must be in the form/service module because it is part of your business constraints. It should also be in the view because you should only present actions that users can actually execute.

[Aryeh Leib Taurog]

Custom manager methods are a good way to implement queries: User.objects.inactive_users(). But the proxy model example here IMO leads to incorrect semantics: u = InactiveUser.objects.all()[0]; u.active = True; u.save() and yet isinstance(u, InactiveUser) == True. Also I would mention an effective way to maintain a query model in many cases is with a db view.

[Teimuraz]

I like this approach too, coming also fro java. I'm new to python, How would you test views.py? How would you mock service layer (if, for example, service makes some remote api calls) ?

[andho]

Django does have a lot more to offer apart from django.db.models, but most of the ecosystem depends heavily on your model using the django models.

[Mohammad Torkashvand]

The design pattern used for developing software. And django designed to easily be used to deliver software in mid or big scale not just web pages!

[Maks]

Actually i see it as an easy to use approach. But i would move final model to another file, and not inherit in models.py. It would be like service.py were clash userlogic+model

[adnanmuttaleb]

Great answer, but I have to say this is mainly CQRS not DDD.

[publysher]

@adnanmuttaleb This is correct. Note that the answer itself only uses the term "Domain Model". I've included the link to DDD not because my answer is DDD, but because that book does a great job in helping you think about domain models.

[m02ph3u5]

You should add some explanation.

[djvg]

For those interested, there's an FAQ about MVC in the Django docs.

[Jim]

I don't understand how you run the 'execute' method of the ActivateUserForm is execute is not part of the Forms API.

[Rik Schoonbeek]

A book came out this year about applying some architectural patterns (a lot of which seem to be DDD patterns), with Flask & Django used as examples. The title is 'Architecture Patterns with Python'. If you want to learn more in depth what a Domain Model would look like, and how you would combine that with the Django ORM (Repository pattern), and more things like that, I highly recommend this book.

[ImportError]

Good. The services.py module is a similar application of the Data Access Object pattern