Method delegation in python

Question

I'm writing a small framework for orchestrating AWS clusters and there are some common hierarchical patterns that appear over and over again. One such pattern is gathering a collection of instances into a bigger object and then delegating some methods down to all the instances directly. So instead of copying and pasting the same boilerplate code over and over again I abstracted it with the following pattern:

def __getattr__(self, item):
    if not item in self._allowed_items:
        raise NonDelegatableItem

    def delegator():
        for instance in self.all_instances:
            getattr(instance, item)()

    return delegator

Is there a better way or pattern for accomplishing the delegation?

Solution 1

__getattr__ is called when the whole class hirarchy is traversed and the attribute is not found. So it is better to generate the method once and store it in the class. Then finding the method takes less time next time.

>>> X.a

Traceback (most recent call last):
  File "<pyshell#15>", line 1, in <module>
    X.a
AttributeError: class X has no attribute 'a'
>>> x.a
new delegator
<function delegator at 0x02937D30>
>>> x.a
<bound method X.delegator of <__main__.X instance at 0x028DBC60>>
>>> X.a
<unbound method X.delegator>

Here you can see the adaption of your code to do that:

class NonDelegatableItem(AttributeError):
    pass

class X:
    def __getattr__(self, method_name):
        self.check_method_name_is_delegator(method_name)
        return self.create_delegator(method_name)

    def check_method_name_is_delegator(self, method_name):
        if method_name not in self._allowed_items:
            raise NonDelegatableItem('{} can not be delegated'.format(method_name))

    @classmethod
    def create_delegator(cls, method_name):
        print 'new delegator'
        def delegator(self, *args, **kw):
            self.check_method_name_is_delegator(method_name)
            for instance in self.all_instances:
                getattr(instance, method_name)(*args, **kw)
        setattr(cls, method_name, delegator)
        return delegator


x = X()

x._allowed_items = ['a', 'b']

Solution 2

I've been researching on this and found two solutions. Use a decorator to change the class and create the delegators, or using descriptors for the delegators. I started with the first and then evolve to the second which I like more, so I will start by it. Both can be found here: https://gist.github.com/dhilst/7435a09b4419da349bb4cc4ae855a451 with doctests :)

-- Edit --

For anybody interested I made this a library: https://pypi.org/project/delegateto/

There were bugs in gist implementation, people contributed to this on github, the pypi project is updated, gist not. I strongly recommend you to use pypi version.

Using descriptors

Descriptors are things that can be getted and setted. In this case we are interested in the gettable ability of descriptors. The delegate descriptor defined like this

class DelegateTo:
    def __init__(self, to, method=None):
        self.to = to
        self.method = method
    def __get__(self, obj, objecttype):
        if self.method is not None:
            return getattr(getattr(obj, self.to), self.method)

        for method, v in obj.__class__.__dict__.items():
            if v is self:
                self.method = method
                return getattr(getattr(obj, self.to), method)

And is used like this

class Foo:
    upper = DelegateTo('v')
    __len__ = DelegateTo('l')
    __iter__ = DelegateTo('l')
    def __init__(self, v, l):
         self.v = v
         self.l = l

To call a descriptor simply call the method Foo('hello').upper(). Magic methods also works len(Foo('', [1,2,3,4])) returns 4. The gist link above has a more powerful implementation but the basics is the same.

Using decorators

Everytime that you need to change a class behavior in an repetitive way, a decorator is a candidate. In this case the decorator will call setattr at the class to create the delegators.

def delegate(to, *methods):
    def dec(klass):
        def create_delegator(method):
            def delegator(self, *args, **kwargs):
                obj = getattr(self, to)
                m = getattr(obj, method)
                return m(*args, **kwargs)
            return delegator
        for m in methods:
            setattr(klass, m, create_delegator(m))
        return klass
    return dec

The usage is also simple, just decorate the class, as many times you want. The decorator will modify the class inplace so the same class is returned.

Here is a usage

@delegate('v', 'upper', 'lower')
class Foo:
   def __init__(self, v):
       self.v = v

And the call of the delegated method is also transparent Foo('hello').upper(). I prefer the second one because it seems more idiomatic for me. The decorator has an advantage to support multiple methods but this is implementable on the descriptor form too.

Again, I really recommend that you see the gist: https://gist.github.com/dhilst/7435a09b4419da349bb4cc4ae855a451 there are tons of examples in the docstring. Just modify them and execute the scripts to play around.

-- Edit --

For any body interested, I make this a pip package https://pypi.org/project/delegateto/

-- Edit --

There were bugs in gist implementation, people contributed to this on github, the pypi project is updated, gist not. I strongly recommend you to use pypi version.

Regards

Comments

[David K.]

Good point. I didn't think of assigning the method as well along the way.

[Tobias Kienzler]

Nice one. Shouldn't __getattr__'s call to check_method_name_is_delegator be omitted so a non-delegatable attribute is not looked up again (someone could catch the NonDelegatableItem via an except AttributeError after all)? And the second call could be put outside the delegator with an except NonDelegatableItem: def delegator(self, *args, **kw): raise NonDelegatableItem(... so the delegatability is only checked once.

[User]

@TobiasKienzler Sounds reasonable. You can create a new answer with this. We can discuss it afterwards.

[Tobias Kienzler]

@User Hm, StackOverflow should have a github-ish "fork answer" option ;) Actually I spoke to soon: your setattr(cls, method_name, delegator) already takes care of preventing another call to __getattr__ (I confused it with the always-called __getattribute__), and by splitting off the check_method_name_is_delegator-check in def delegator one takes away the option to dynamically modify the _allowed_items. So the only remaining suggestion would be omitting __getattr__'s call to check_method_name_is_delegator, though that slightly increases performance for non-delegators...