In Python, how do I indicate I'm overriding a method?

Question

In Java, for example, the @Override annotation not only provides compile-time checking of an override but makes for excellent self-documenting code.

I'm just looking for documentation (although if it's an indicator to some checker like pylint, that's a bonus). I can add a comment or docstring somewhere, but what is the idiomatic way to indicate an override in Python?

Solution 1

Based on this and fwc:s answer I created a pip installable package https://github.com/mkorpela/overrides

From time to time I end up here looking at this question. Mainly this happens after (again) seeing the same bug in our code base: Someone has forgotten some "interface" implementing class while renaming a method in the "interface"..

Well Python ain't Java but Python has power -- and explicit is better than implicit -- and there are real concrete cases in the real world where this thing would have helped me.

So here is a sketch of overrides decorator. This will check that the class given as a parameter has the same method (or something) name as the method being decorated.

If you can think of a better solution please post it here!

def overrides(interface_class):
    def overrider(method):
        assert(method.__name__ in dir(interface_class))
        return method
    return overrider

It works as follows:

class MySuperInterface(object):
    def my_method(self):
        print 'hello world!'


class ConcreteImplementer(MySuperInterface):
    @overrides(MySuperInterface)
    def my_method(self):
        print 'hello kitty!'

and if you do a faulty version it will raise an assertion error during class loading:

class ConcreteFaultyImplementer(MySuperInterface):
    @overrides(MySuperInterface)
    def your_method(self):
        print 'bye bye!'

>> AssertionError!!!!!!!

Solution 2

Here's an implementation that doesn't require specification of the interface_class name.

import inspect
import re

def overrides(method):
    # actually can't do this because a method is really just a function while inside a class def'n  
    #assert(inspect.ismethod(method))

    stack = inspect.stack()
    base_classes = re.search(r'class.+\((.+)\)\s*\:', stack[2][4][0]).group(1)

    # handle multiple inheritance
    base_classes = [s.strip() for s in base_classes.split(',')]
    if not base_classes:
        raise ValueError('overrides decorator: unable to determine base class') 

    # stack[0]=overrides, stack[1]=inside class def'n, stack[2]=outside class def'n
    derived_class_locals = stack[2][0].f_locals

    # replace each class name in base_classes with the actual class type
    for i, base_class in enumerate(base_classes):

        if '.' not in base_class:
            base_classes[i] = derived_class_locals[base_class]

        else:
            components = base_class.split('.')

            # obj is either a module or a class
            obj = derived_class_locals[components[0]]

            for c in components[1:]:
                assert(inspect.ismodule(obj) or inspect.isclass(obj))
                obj = getattr(obj, c)

            base_classes[i] = obj


    assert( any( hasattr(cls, method.__name__) for cls in base_classes ) )
    return method

Solution 3

If you want this for documentation purposes only, you can define your own override decorator:

def override(f):
    return f


class MyClass (BaseClass):

    @override
    def method(self):
        pass

This is really nothing but eye-candy, unless you create override(f) in such a way that is actually checks for an override.

But then, this is Python, why write it like it was Java?

Solution 4

Improvising on @mkorpela great answer, here is a version with

more precise checks, naming, and raised Error objects

def overrides(interface_class):
    """
    Function override annotation.
    Corollary to @abc.abstractmethod where the override is not of an
    abstractmethod.
    Modified from answer https://stackoverflow.com/a/8313042/471376
    """
    def confirm_override(method):
        if method.__name__ not in dir(interface_class):
            raise NotImplementedError('function "%s" is an @override but that'
                                      ' function is not implemented in base'
                                      ' class %s'
                                      % (method.__name__,
                                         interface_class)
                                      )

        def func():
            pass

        attr = getattr(interface_class, method.__name__)
        if type(attr) is not type(func):
            raise NotImplementedError('function "%s" is an @override'
                                      ' but that is implemented as type %s'
                                      ' in base class %s, expected implemented'
                                      ' type %s'
                                      % (method.__name__,
                                         type(attr),
                                         interface_class,
                                         type(func))
                                      )
        return method
    return confirm_override

Here is what it looks like in practice:

NotImplementedError "not implemented in base class"

class A(object):
    # ERROR: `a` is not a implemented!
    pass

class B(A):
    @overrides(A)
    def a(self):
        pass

results in more descriptive NotImplementedError error

function "a" is an @override but that function is not implemented in base class <class '__main__.A'>

full stack

Traceback (most recent call last):
  …
  File "C:/Users/user1/project.py", line 135, in <module>
    class B(A):
  File "C:/Users/user1/project.py", line 136, in B
    @overrides(A)
  File "C:/Users/user1/project.py", line 110, in confirm_override
    interface_class)
NotImplementedError: function "a" is an @override but that function is not implemented in base class <class '__main__.A'>

NotImplementedError "expected implemented type"

class A(object):
    # ERROR: `a` is not a function!
    a = ''

class B(A):
    @overrides(A)
    def a(self):
        pass

results in more descriptive NotImplementedError error

function "a" is an @override but that is implemented as type <class 'str'> in base class <class '__main__.A'>, expected implemented type <class 'function'>

full stack

Traceback (most recent call last):
  …
  File "C:/Users/user1/project.py", line 135, in <module>
    class B(A):
  File "C:/Users/user1/project.py", line 136, in B
    @overrides(A)
  File "C:/Users/user1/project.py", line 125, in confirm_override
    type(func))
NotImplementedError: function "a" is an @override but that is implemented as type <class 'str'> in base class <class '__main__.A'>, expected implemented type <class 'function'>

---

The great thing about @mkorpela answer is the check happens during some initialization phase. The check does not need to be "run". Referring to the prior examples, class B is never initialized (B()) yet the NotImplementedError will still raise. This means overrides errors are caught sooner.

Solution 5

Python ain't Java. There's of course no such thing really as compile-time checking.

I think a comment in the docstring is plenty. This allows any user of your method to type help(obj.method) and see that the method is an override.

You can also explicitly extend an interface with class Foo(Interface), which will allow users to type help(Interface.method) to get an idea about the functionality your method is intended to provide.

View more solutions

Comments

[Bluu]

In other words, you don't ever indicate that you're overriding a method? Leave it to the reader to figure that out himself?

[Ed S.]

Yeah, I know it seems like an error prone situation coming from a compiled language, but you just have to accept it. In practice I have not found it to be much of a problem (Ruby in my case, not Python, but same idea)

[Bluu]

Sure, done. Both Triptych's answer and mkorpela's answers are simple, I like that, but the latter's explicit-over-implicit spirit, and intelligibly preventing mistakes wins.

[letmaik]

It's not directly the same, but abstract base classes check if all abstract methods have been overridden by a subclass. Of course this doesn't help if you're overriding concrete methods.

[Pavel Minaev]

The real point of @Override in Java isn't to document - it's to catch a mistake when you intended to override a method, but ended up defining a new one (e.g. because you misspelled a name; in Java, it may also happen because you used the wrong signature, but this isn't an issue in Python - but spelling mistake still is).

[Erik Kaplun]

One could add actual validation via inspection to the override decorator.

[Tuukka Mustonen]

@ Pavel Minaev: True, but it is still convenient to have for documentation, especially if you're using an IDE / text editor which doesn't have automatic indicators for overrides (Eclipse's JDT shows them neatly alongside line numbers, for example).

[siamii]

@PavelMinaev Wrong. One of the main points of @Override is documentation in addition to compile time checking.

[Christopher Bruns]

Awesome. This caught a misspelling bug the first time I tried it. Kudos.

[Hamish Grubijan]

So, I am using python 2.7 and if my class extends a bunch of other classes and, unlike with the interface, I do not want to hard-code the exact class name that contains the interface function, then can this work in general if I inherit from more than one class or will the method resolution order break this?

[Skarab]

How would you write an assertion that checks whether the number of arguments is the same in the base class/interface and the child class?

[Piotr Dobrogost]

But then, this is Python, why write it like it was Java? Because some ideas in Java are good and worth extending to other languages?

[Abgan]

Because when you rename a method in a superclass it would be nice to know that some subclass 2 levels down was overriding it. Sure, it's easy to check, but a little help from language parser wouldn't hurt.

[mfbutner]

I like your decorator, but its too bad that the assertion has to be called every time the function is executed. Might there be some way of getting around this using metaclasses?

[Bluu]

A little magical but makes typical usage a lot easier. Can you include usage examples?

[larham1]

what are the average and worst-case costs of using this decorator, perhaps expressed as a comparison with a build-in decorator like @classmethod or @property?

[mkorpela]

mfbutner: it is not called every time the method is executed - only when the method is created.

[Abgan]

@larham1 This decorator is executed once, when class definition is analyzed, not on each call. Therefore it's execution cost is irrelevant, when compared to program runtime.

[letmaik]

This is also nice for doc strings! overrides could copy the docstring of the overridden method if the overriding method doesn't have one of its own.

[Andrew Mellinger]

@siamii I think an aid to documentation is great, but in all the official Java documentation I see, they only indicate the importance of the compile time checks. Please substantiate your claim that Pavel is "wrong."

[Admin]

@mkorpela, Heh this code of yours should be in python default lib system. Why you don't put this in pip system? :P

[mkorpela]

@urosjarc: great idea! I thought it might be too little code for a pip package but why not.

[Admin]

@mkorpela : Oh and I suggest to inform the python core developers obout this package, they might want to consider obout adding overide decorator to the core python system. :)

[Neil G]

If the overridden method is marked with @abstractmethod, then you could also check for an identical parameter list.

[Neil G]

This will be much nicer in Python 3.6 thanks to PEP 487.

[Xiong Chiamiov]

To note: the Python package is quite more sophisticated than the original sketch, and no longer requires re-specifying the base class.

[sfkleach]

Because it is a good idea. The fact that a variety of other languages have the feature is no argument - either for or against.

[Neil G]

Hey! This looks interesting. Could you consider doing a pull request on my ipromise project? I've added an answer.

[JamesThomasMoon]

@NeilG I forked the ipromise project and coded a bit. It seems like you've essentially implemented this within overrides.py. I'm not sure what else I can significantly improve except to change the exception types from TypeError to NotImplementedError.

[Neil G]

Hey! Thanks, I don't have checks that the overridden object actually has type types.MethodType. That was a good idea in your answer.

[Ivan Kovtun]

To get better error message: assert any(hasattr(cls, method.__name__) for cls in base_classes), 'Overriden method "{}" was not found in the base class.'.format(method.__name__)

[Admin]

@mkorpela I just added your library to my project and it works perfect! Thanks and job well done.

[Bluu]

Got an example? Something that would pass/fail mypy?

[Janus Troelsen]

@Bluu: Yes, look at the section defining a protocol

[Bluu]

Can you include example source code and mypy results in your answer? And required version number and tools (I think mypy is new to this question thread)? I think that would make your answer more self-contained.

[Janus Troelsen]

@Bluu I have added it.

[Admin]

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.