Why Python decorators rather than closures?

Question

I still haven't got my head around decorators in Python.

I've already started using a lot of closures to do things like customize functions and classes in my coding.

Eg.

class Node :
    def __init__(self,val,children) :
        self.val = val
        self.children = children

def makeRunner(f) :
    def run(node) :
        f(node)
        for x in node.children :
            run(x)
    return run

tree=Node(1,[Node(2,[]),Node(3,[Node(4,[]),Node(5,[])])])

def pp(n) : print "%s," % n.val
printTree = makeRunner(pp)
printTree(tree)

As far as I can see, decorators are just a different syntax for doing something similar.

Instead of

def pp(n) : print "%s," % n.val
printTree = makeRunner(pp)

I would write :

@makeRunner
def printTree(n) : print "%s," % n.val

Is this all there is to decorators? Or is there a fundamental difference that I've missed?

Solution 1

While it is true that syntactically, decorators are just "sugar", that is not the best way to think about them.

Decorators allow you to weave functionality into your existing code without actually modifying it. And they allow you to do it in a way that is declarative.

This allows you to use decorators to do aspect-oriented programming (AOP). So you want to use a decorator when you have a cross-cutting concern that you want to encapsulate in one place.

The quintessential example would probably be logging, where you want to log the entry or exit of a function, or both. Using a decorator is equivalent to applying advice (log this!) to a joinpoint (during method entry or exit).

Method decoration is a concept like OOP or list comprehensions. As you point out, it is not always appropriate, and can be overused. But in the right place, it can be useful for making code more modular and decoupled.

Solution 2

Are your examples real code, or just examples?

If they're real code, I think you overuse decorators, probably because of your background (i.e. you are used to other programming languages)

Stage 1: avoiding decorators

def run(rootnode, func):
    def _run(node): # recursive internal function
        func(node)
        for x in node.children:
            _run(x) # recurse
    _run(rootnode) # initial run

This run method obsoletes makeRunner. Your example turns to:

def pp(n): print "%s," % n.val
run(tree, pp)

However, this ignores completely generators, so…

Stage 2: using generators

class Node :
    def __init__(self,val,children) :
        self.val = val
        self.children = children

    def __iter__(self): # recursive
        yield self
        for child in self.children:
            for item in child: # recurse
                yield item

def run(rootnode, func):
    for node in rootnode:
        func(node)

Your example remains

def pp(n): print "%s," % n.val
run(tree, pp)

Note that the special method __iter__ allows us to use the for node in rootnode: construct. If you don't like it, just rename the __iter__ method to e.g. walker, and change the run loop into: for node in rootnode.walker():
Obviously, the run function could be a method of class Node instead.

As you see, I suggest you use directly run(tree, func) instead of binding them to the name printTree, but you can use them in a decorator, or you can make use of the functools.partial function:

printTree= functools.partial(run, func=pp)

and from then on, you would just

printTree(tree)

Solution 3

Decorators, in the general sense, are functions or classes that wrap around another object, that extend, or decorate the object. The decorator supports the same interface as the wrapped function or object, so the receiver doesn't even know the object has been decorated.

A closure is an anonymous function that refers to its parameters or other variables outside its scope.

So basically, decorators uses closures, and not replace them.

def increment(x):
    return x + 1

def double_increment(func):
    def wrapper(x):
        print 'decorator executed'
        r = func(x)   # --> func is saved in __closure__
        y = r * 2
        return r, y
    return wrapper

@double_increment
def increment(x):
    return x + 1

>>> increment(2)
decorator executed
(3, 6)

>>> increment.__closure__
(<cell at 0x02C7DC50: function object at 0x02C85DB0>,)

>>> increment.__closure__[0].cell_contents 
<function increment at 0x02C85DB0>

So the decorator saves the original function with closure.

Solution 4

Following up Dutch Master's AOP reference, you'll find that using decorators becomes especially useful when you start adding parameters to modify the behaviour of the decorated function/method, and reading that above the function definition is so much easier.

In one project I recall, we needed to supervise tons of celery tasks and so we came up with the idea of using a decorator to plug-and-tweak as required, which was something like:

class tracked_with(object):
    """
    Method decorator used to track the results of celery tasks.
    """
    def __init__(self, model, unique=False, id_attr='results_id',
                 log_error=False, raise_error=False):
        self.model = model
        self.unique = unique
        self.id_attr = id_attr
        self.log_error = log_error
        self.raise_error = raise_error

    def __call__(self, fn):

        def wrapped(*args, **kwargs):
            # Unique passed by parameter has priority above the decorator def
            unique = kwargs.get('unique', None)
            if unique is not None:
                self.unique = unique

            if self.unique:
                caller = args[0]
                pending = self.model.objects.filter(
                    state=self.model.Running,
                    task_type=caller.__class__.__name__
                )
                if pending.exists():
                    raise AssertionError('Another {} task is already running'
                                         ''.format(caller.__class__.__name__))

            results_id = kwargs.get(self.id_attr)
            try:
                result = fn(*args, **kwargs)

            except Retry:
                # Retry must always be raised to retry a task
                raise

            except Exception as e:
                # Error, update stats, log/raise/return depending on values
                if results_id:
                    self.model.update_stats(results_id, error=e)
                if self.log_error:
                    logger.error(e)
                if self.raise_error:
                    raise
                else:
                    return e

            else:
                # No error, save results in refresh object and return
                if results_id:
                    self.model.update_stats(results_id, **result)
                return result

        return wrapped

Then we simply decorated the run method on the tasks with the params required for each case, like:

class SomeTask(Task):

    @tracked_with(RefreshResults, unique=True, log_error=False)
    def run(self, *args, **kwargs)...

Then changing the behaviour of the task (or removing the tracking altogether) meant tweaking one param, or commenting out the decorated line. Super easy to implement, but more importantly, super easy to understand on inspection.

View more solutions