Why is Python 3.x's super() magic?

Question

In Python 3.x, super() can be called without arguments:

class A(object):
    def x(self):
         print("Hey now")

class B(A):
    def x(self):
        super().x()

>>> B().x()
Hey now

In order to make this work, some compile-time magic is performed, one consequence of which is that the following code (which rebinds super to super_) fails:

super_ = super

class A(object):
    def x(self):
        print("No flipping")

class B(A):
    def x(self):
        super_().x()

>>> B().x()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in x
RuntimeError: super(): __class__ cell not found

Why is super() unable to resolve the superclass at runtime without assistance from the compiler? Are there practical situations in which this behaviour, or the underlying reason for it, could bite an unwary programmer?

... and, as a side question: are there any other examples in Python of functions, methods etc. which can be broken by rebinding them to a different name?

Solution

The new magic super() behaviour was added to avoid violating the D.R.Y. (Don't Repeat Yourself) principle, see PEP 3135. Having to explicitly name the class by referencing it as a global is also prone to the same rebinding issues you discovered with super() itself:

class Foo(Bar):
    def baz(self):
        return super(Foo, self).baz() + 42

Spam = Foo
Foo = something_else()

Spam().baz()  # liable to blow up

The same applies to using class decorators where the decorator returns a new object, which rebinds the class name:

@class_decorator_returning_new_class
class Foo(Bar):
    def baz(self):
        # Now `Foo` is a *different class*
        return super(Foo, self).baz() + 42

The magic super() __class__ cell sidesteps these issues nicely by giving you access to the original class object.

The PEP was kicked off by Guido, who initially envisioned super becoming a keyword, and the idea of using a cell to look up the current class was also his. Certainly, the idea to make it a keyword was part of the first draft of the PEP.

However, it was in fact Guido himself who then stepped away from the keyword idea as 'too magical', proposing the current implementation instead. He anticipated that using a different name for super() could be a problem:

My patch uses an intermediate solution: it assumes you need __class__ whenever you use a variable named 'super'. Thus, if you (globally) rename super to supper and use supper but not super, it won't work without arguments (but it will still work if you pass it either __class__ or the actual class object); if you have an unrelated variable named super, things will work but the method will use the slightly slower call path used for cell variables.

So, in the end, it was Guido himself that proclaimed that using a super keyword did not feel right, and that providing a magic __class__ cell was an acceptable compromise.

I agree that the magic, implicit behaviour of the implementation is somewhat surprising, but super() is one of the most mis-applied functions in the language. Just take a look at all the misapplied super(type(self), self) or super(self.__class__, self) invocations found on the Internet; if any of that code was ever called from a derived class you'd end up with an infinite recursion exception. At the very least the simplified super() call, without arguments, avoids that problem.

As for the renamed super_; just reference __class__ in your method as well and it'll work again. The cell is created if you reference either the super or __class__ names in your method:

>>> super_ = super
>>> class A(object):
...     def x(self):
...         print("No flipping")
... 
>>> class B(A):
...     def x(self):
...         __class__  # just referencing it is enough
...         super_().x()
... 
>>> B().x()
No flipping

Comments

[Games Brainiac]

I'll let Armin do the explaining on this one. This is also another good post

[wim]

related: stackoverflow.com/q/36993577/674039

[Charles Merriam]

Good write-up. It's still as clear as mud however. You are saying that super() is equivalent to an automatically instantiated function like def super(of_class=magic __class__) kind of like a self.super(); def super(self): return self.__class__?

[Martijn Pieters]

@CharlesMerriam: This post is not about how super() without arguments works; it mostly deals with the why it exists. super(), in a class method, is equivalent to super(ReferenceToClassMethodIsBeingDefinedIn, self), where ReferenceToClassMethodIsBeingDefinedIn is determined at compile time, attached to the method as a closure named __class__, and super() will look up both from the calling frame at runtime. But you don't actually need to know all this.

[Martijn Pieters]

@CharlesMerriam: but super() is nowhere close to being an automatically instantiated function, no.

[chris]

I just opened a similar question at stackoverflow.com/q/48691652/1236579 that was closed as an exact duplicate. However, I don't understand how it can be considered a duplicate for reasons I included in an update and a comment for that post. It could be that I don't understand something. Can someone read my question, comment, and update empathetically and either tell me what I'm missing or validate that it shouldn't really be a dupe?

[Martijn Pieters]

@chris.leonard: the key sentence is The cell is created if you use super() or use __class__ in your method. You used the name super in your function. The compiler sees that and adds the __class__ closure.

[chris]

@MartijnPieters: I see. I did not understand the bit about the cell. Now I know what to read up on and can make progress. Thank you all for helping with what was, in hindsight, an unnecessary question.

[Alexey]

"Having to explicitly name the class by referencing it as a global" -- but why? It is enough to use type(self), as everywhere else.

[Martijn Pieters]

@Alexey: it is not enough. type(self) gives the current type, which is not the same as the type the method is defined on. So a class Foo with method baz needs super(Foo, self).baz(), because it could be subclassed as class Ham(Foo):, at which point type(self) is Ham and super(type(self), self).baz() would give you an infinite loop. See the post I link to in my answer: When calling super() in a derived class, can I pass in self.__class__?

[Alexey]

This seems ok to me, this is what inheritance does. In this example, isn't super(type(self), self).baz() exactly what you want to call?

[Martijn Pieters]

@Alexey: it only looks enough if you don't subclass. That's deceptive, and a common error to make because it only blows up when you do subclass. Try out the code in the other post I linked and you'll see why you can't use type(self).