Decorators with parameters?

Question

I have a problem with the transfer of the variable insurance_mode by the decorator. I would do it by the following decorator statement:

@execute_complete_reservation(True)
def test_booking_gta_object(self):
    self.test_select_gta_object()

but unfortunately, this statement does not work. Perhaps maybe there is better way to solve this problem.

def execute_complete_reservation(test_case,insurance_mode):
    def inner_function(self,*args,**kwargs):
        self.test_create_qsf_query()
        test_case(self,*args,**kwargs)
        self.test_select_room_option()
        if insurance_mode:
            self.test_accept_insurance_crosseling()
        else:
            self.test_decline_insurance_crosseling()
        self.test_configure_pax_details()
        self.test_configure_payer_details

    return inner_function

Solution 1

The syntax for decorators with arguments is a bit different - the decorator with arguments should return a function that will take a function and return another function. So it should really return a normal decorator. A bit confusing, right? What I mean is:

def decorator_factory(argument):
    def decorator(function):
        def wrapper(*args, **kwargs):
            funny_stuff()
            something_with_argument(argument)
            result = function(*args, **kwargs)
            more_funny_stuff()
            return result
        return wrapper
    return decorator

Here you can read more on the subject - it's also possible to implement this using callable objects and that is also explained there.

Solution 2

Edit : for an in-depth understanding of the mental model of decorators, take a look at this awesome Pycon Talk. well worth the 30 minutes.

One way of thinking about decorators with arguments is

@decorator
def foo(*args, **kwargs):
    pass

translates to

foo = decorator(foo)

So if the decorator had arguments,

@decorator_with_args(arg)
def foo(*args, **kwargs):
    pass

translates to

foo = decorator_with_args(arg)(foo)

decorator_with_args is a function which accepts a custom argument and which returns the actual decorator (that will be applied to the decorated function).

I use a simple trick with partials to make my decorators easy

from functools import partial

def _pseudo_decor(fun, argument):
    def ret_fun(*args, **kwargs):
        #do stuff here, for eg.
        print ("decorator arg is %s" % str(argument))
        return fun(*args, **kwargs)
    return ret_fun

real_decorator = partial(_pseudo_decor, argument=arg)

@real_decorator
def foo(*args, **kwargs):
    pass

Update:

Above, foo becomes real_decorator(foo)

One effect of decorating a function is that the name foo is overridden upon decorator declaration. foo is "overridden" by whatever is returned by real_decorator. In this case, a new function object.

All of foo's metadata is overridden, notably docstring and function name.

>>> print(foo)
<function _pseudo_decor.<locals>.ret_fun at 0x10666a2f0>

functools.wraps gives us a convenient method to "lift" the docstring and name to the returned function.

from functools import partial, wraps

def _pseudo_decor(fun, argument):
    # magic sauce to lift the name and doc of the function
    @wraps(fun)
    def ret_fun(*args, **kwargs):
        # pre function execution stuff here, for eg.
        print("decorator argument is %s" % str(argument))
        returned_value =  fun(*args, **kwargs)
        # post execution stuff here, for eg.
        print("returned value is %s" % returned_value)
        return returned_value

    return ret_fun

real_decorator1 = partial(_pseudo_decor, argument="some_arg")
real_decorator2 = partial(_pseudo_decor, argument="some_other_arg")

@real_decorator1
def bar(*args, **kwargs):
    pass

>>> print(bar)
<function __main__.bar(*args, **kwargs)>

>>> bar(1,2,3, k="v", x="z")
decorator argument is some_arg
returned value is None

Solution 3

Here is a slightly modified version of t.dubrownik's answer. Why?

1. As a general template, you should return the return value from the original function.
2. This changes the name of the function, which could affect other decorators / code.

So use @functools.wraps():

from functools import wraps

def create_decorator(argument):
    def decorator(function):
        @wraps(function)
        def wrapper(*args, **kwargs):
            funny_stuff()
            something_with_argument(argument)
            retval = function(*args, **kwargs)
            more_funny_stuff()
            return retval
        return wrapper
    return decorator

Solution 4

I'd like to show an idea which is IMHO quite elegant. The solution proposed by t.dubrownik shows a pattern which is always the same: you need the three-layered wrapper regardless of what the decorator does.

So I thought this is a job for a meta-decorator, that is, a decorator for decorators. As a decorator is a function, it actually works as a regular decorator with arguments:

def parametrized(dec):
    def layer(*args, **kwargs):
        def repl(f):
            return dec(f, *args, **kwargs)
        return repl
    return layer

This can be applied to a regular decorator in order to add parameters. So for instance, say we have the decorator which doubles the result of a function:

def double(f):
    def aux(*xs, **kws):
        return 2 * f(*xs, **kws)
    return aux

@double
def function(a):
    return 10 + a

print function(3)    # Prints 26, namely 2 * (10 + 3)

With @parametrized we can build a generic @multiply decorator having a parameter

@parametrized
def multiply(f, n):
    def aux(*xs, **kws):
        return n * f(*xs, **kws)
    return aux

@multiply(2)
def function(a):
    return 10 + a

print function(3)    # Prints 26

@multiply(3)
def function_again(a):
    return 10 + a

print function(3)          # Keeps printing 26
print function_again(3)    # Prints 39, namely 3 * (10 + 3)

Conventionally the first parameter of a parametrized decorator is the function, while the remaining arguments will correspond to the parameter of the parametrized decorator.

An interesting usage example could be a type-safe assertive decorator:

import itertools as it

@parametrized
def types(f, *types):
    def rep(*args):
        for a, t, n in zip(args, types, it.count()):
            if type(a) is not t:
                raise TypeError('Value %d has not type %s. %s instead' %
                    (n, t, type(a))
                )
        return f(*args)
    return rep

@types(str, int)  # arg1 is str, arg2 is int
def string_multiply(text, times):
    return text * times

print(string_multiply('hello', 3))    # Prints hellohellohello
print(string_multiply(3, 3))          # Fails miserably with TypeError

A final note: here I'm not using functools.wraps for the wrapper functions, but I would recommend using it all the times.

Solution 5

I presume your problem is passing arguments to your decorator. This is a little tricky and not straightforward.

Here's an example of how to do this:

class MyDec(object):
    def __init__(self,flag):
        self.flag = flag
    def __call__(self, original_func):
        decorator_self = self
        def wrappee( *args, **kwargs):
            print 'in decorator before wrapee with flag ',decorator_self.flag
            original_func(*args,**kwargs)
            print 'in decorator after wrapee with flag ',decorator_self.flag
        return wrappee

@MyDec('foo de fa fa')
def bar(a,b,c):
    print 'in bar',a,b,c

bar('x','y','z')

Prints:

in decorator before wrapee with flag  foo de fa fa
in bar x y z
in decorator after wrapee with flag  foo de fa fa

See Bruce Eckel's article for more details.

View more solutions

Comments

[Brian Clapper]

Your example is not syntactically valid. execute_complete_reservation takes two parameters, but you're passing it one. Decorators are just syntactic sugar for wrapping functions inside other functions. See docs.python.org/reference/compound_stmts.html#function for complete documentation.

[Admin]

Beware of decorator classes. They don't work on methods unless you manually reinvent the logic of instancemethod descriptors.

[Ross Rogers]

delnan, care to elaborate? I've only had to use this pattern once, so I haven't hit any of the pitfalls yet.

[falek.marcin]

Thanks, your solution is more suited to my problem - Easily explains how to create decorators with the parameters

[Alois Mahdal]

I just did this with lambdas all over the place. (read: Python is awesome!) :)

[jamesc]

@RossRogers My guess is that @delnan is referring to things like __name__ which an instance of the decorator class won't have?

[Admin]

@jamesc That too, though that's relatively easy to solve. The specific case I was referring to was class Foo: @MyDec(...) def method(self, ...): blah which does not work because Foo().method won't be a bound method and won't pass self automatically. This too can be fixed, by making MyDec a descriptor and creating bound methods in __get__, but it's more involved and much less obvious. In the end, decorator classes are not as convenient as they seem.

[Michel Müller]

I wonder why GVR didn't implement it by passing in the parameters as subsequent decorator arguments after 'function'. 'Yo dawg I heard you like closures...' etcetera.

[senderle]

@MichelMüller, interesting! But it occurs to me that you'd need to adopt a convention. Would function be the first argument or last? I could make cases for either. The choice seems arbitrary, and so you'd have another random thing to memorize about the language. It's also weird that you'd "call" the function with a signature different from the one in the definition. I suppose that also happens with class methods -- but classes are full-blown language constructs with deep semantics, whereas decorators are just supposed to be syntactic sugar.

[Michel Müller]

> Would function be the first argument or last? Obviously first, since the parameters is a parameter list of variable length. > It's also weird that you'd "call" the function with a signature different from the one in the definition. As you point out, it would fit pretty well actually - it's pretty much analogous to how a class method is called. To make it more clear, you could have something like decorator(self_func, param1, ...) convention. But note: I'm not advocating for any change here, Python is too far down the road for that and we can see how breaking changes have worked out..

[socketpair]

you forgot VERY USEFUL functools.wraps for decorating wrapper :)

[Raffi]

It works but I am sometimes having problems with argument that is not seen in the inner wrapper function (2.7.6, osx apple version). Do you know if this issue is a known issue?

[formiaczek]

You forgot about return when calling function, i.e. return function(*args, **kwargs)

[HaPsantran]

@delnan I'd like to see this caveat featured more prominently. I'm hitting it and am interested in seeing a solution that DOES work (more involved an less obvious though it may be).

[mouckatron]

Didn't use this exactly, but helped me get my head around the concept :) Thanks!

[Jeff]

I tried this and had some issues.

[Dacav]

@Jeff could you share with us the kind of issues you had?

[Jeff]

I had it linked on my question, and I did figure it out... I needed to call @wraps in mine for my particular case.

[Dacav]

Aha! I suspected that, as it was the only possible fault that I could spot. Indeed I changed my answer and appended a mention about wraps. True story :)

[Patrick Mevzek]

Maybe obvious, but just in case: you need to use this decorator as @decorator() and not just @decorator, even if you have only optional arguments.

[zsf222]

Your answer perfectly explained the inherent orthogonality of the decorator, thank you

[Mohammad Banisaeid]

To make this complete, you may also want to add @functools.wraps(function) above the def wrapper... line.

[z33k]

It works but I have trouble getting my head around it. How would one translate the decorators' syntactic sugar into mundane calls in that case? Keeping with your example would it be: function = parametrized(multiply(function, 2)) and then calling function(3) to get 26? It doesn't work when I try it with my analogues

[Dacav]

@o'rety I'm a bit confused by your question. Could you elaborate?

[z33k]

Oh boy, I lost a whole day on this. Thankfully, I came around this answer (which incidentally could be the best answer ever created on the whole internet). They too use your @parametrized trick. The problem I had was I forgot the @ syntax equals actual calls (somehow I knew that and didn't know that at the same time as you can gather from my question). So if you want to translate @ syntax into mundane calls to check how it works, you better comment it out temporarily first or you'd end up calling it twice and getting mumbojumbo results

[Mr_and_Mrs_D]

Could you add @functools.wraps?

[srj]

@Mr_and_Mrs_D , I've updated the post with an example with functool.wraps. Adding it in the example may confuse readers further.

[Stefan Falk]

What is arg here!?

[srj]

@StefanFalk arg is just a variable name, with the value that you'd use for creating the real_decorator out of _pseudo_decor

[Chang Zhao]

How will you pass argument passed to bar to the argument of real_decorator ?

[srj]

@ChangZhao as described above, even though you use the name bar in the code after the declaration, python "points" the name bar to the result of real_decorator(bar) so any arguments that you pass while calling bar(arg1, arg2..) is in fact interpreted as real_decorator(bar)(arg1, arg2,..)

[Chang Zhao]

how will that work with partial ? because you are passing arg !

[srj]

Try it out for yourself. hopefully this piece of code is self-explanatory. repl.it/@sreedom/SpotlessTastyScreencast

[Gyula Sámuel Karli]

It's cool on one hand but I don't understand the reason for having 3 methods encapsulating to each other. Namely I don't understand what on the earth repl() does, is it just a trick with scopes and param numbers?

[Dacav]

repl stands for replacement. When I name it repl it means that it will effectively take the place of another function. Assuming that you are talking about the parametrized decorator, layer will replace the decorated decorator. The dec parameter of parametrized is the decorator we are going to replace with layer. Decorators take functions as parameter, and indeed we pass f to dec. So, ultimately, repl is what replaces dec, and works by applying the decorator dec so that it replaces f. I hope that this comment makes it clearer, but I myself have to read it many times.

[Arindam Roychowdhury]

This works. Although yes, this makes it hard to set the value to the decorator.

[Nicholas Humphrey]

Thanks, years late but this solved my problem. Finally I'm feeling that I'm grasping the idea of decorators.

[Cho]

The nicest explanation I've ever seen!

[norok2]

Note also that factor_or_func (or any other parameter) should never gets reassigned in wrapper().

[Stephen Ellwood]

This seems like a great answer but there are some steps here I do not understand. Why copy self into decorator_self. Is this to get around the issue mentioned above?

[Ross Rogers]

I don't think the decorator_self is necessary anymore. It was because I had self as the first parameter of wrapee way back when I was experimenting. Try it with out it, just using self and not aliasing self to decorator_self

[Radha Satam]

Thank you for that explanation. The link to the pycon talk was very helpful too.

[Shital Shah]

Why do you need to check in locals()?

[norok2]

@ShitalShah that covers the case where the decorator is used without ().

[Ahmed Gad]

Perfect explanation. This is what every learner to Python decorators is looking for.

[Tian]

Although this is the solution, it really does'nt seem elegant or pythonic. I always viewed decorators as like object methods, which pass in self as an argument automatically when called. Having an additional function layer to handle arguments just does not feel right

[medley56]

This is freaking genius.

[hasdrubal]

So how will it work when you don't know arg until it is time to actually run the function? aka you want to surround a string with an HTML tag but the used tag can be different every time (or even user defined)?

[Lee Loftiss]

The video was great. Thanks!

[Milo Persic]

Of all the posts here, this answer proved the most useful for my understanding of how the argument is passed and handled.

[user8491363]

In numba library, there's @jit(nopython=True) expression. How's that possible then? In this scenario, I don't see how numba is casually just getting an argument.

[srj]

I've edited the example with wraps to be clearer in the use of arg.

[Philip Couling]

@Titan I'd say it's very pythoninc. The nested methods is to allow you to define decorators separate to using them. It makes it possible to have two steps x=decorator_factory(foo) @x; def ... as well as the more common single step @decorator_factory(foo); def .... Though it's less common, it is used and very useful.

[Leon Chang]

This helps me understand how Flask's @app.route(path) maps to or calls its view function. Thanks.

[Lucas Andrade]

I did exactly that, but on AWS lambdas with flask it doesn't work: python 3.8 returns this error: AssertionError: View function mapping is overwriting an existing endpoint function: authorization_required_wrapper

[jouell]

why wouldn't "foo = decorator_with_args(arg)(foo)" be "foo = decorator_with_args(foo(arg))" instead?

[srj]

@jouell, because python will execute the @decorator_with_args(arg) first , the result of that will take in the function object created by the def statement.

[jouell]

Right. Thank you. @srj

[Amer Sawan]

Note that this will not work similiar to the normal decorator, if the any_number_of_arguments is optional arg, you still have to write () in the end of the decorator.

[user41855]

@srj It actually is foo = decorator_with_args(foo)(arg). decorator_with_args(foo) will return the wrapper function. This wrapper function needs 'args' as arguments. Hence wrapper(arg) will be executed next

[nish]

Please expound on " Python expects completely different behavior in these two cases"

[Tommy]

this is my favorite answer here as wraps is critical.

[Jennings]

Thanks for this answer, took me some time to...*wrap* my head around this concept...ba dum tsk... lol. So the key concept here for me was that this is 3 layers deep. Found some more related info: realpython.com/primer-on-python-decorators/…

[chilicheech]

How about changing the multiplying function's return from return _decorator(f_py) if callable(f_py) else _decorator to return _decorator(f_py) if f_py else _decorator or return _decorator if f_py is None else _decorator(f_py) since you've already asserted that it's either a callable or None. This should be "more efficient" than calling callable a second time.