Why does Python use 'magic methods'?

Question

I've been playing around with Python recently, and one thing I'm finding a bit odd is the extensive use of 'magic methods', e.g. to make its length available, an object implements a method, def __len__(self), and then it is called when you write len(obj).

I was just wondering why objects don't simply define a len(self) method and have it called directly as a member of the object, e.g. obj.len()? I'm sure there must be good reasons for Python doing it the way it does, but as a newbie I haven't worked out what they are yet.

Solution 1

AFAIK, len is special in this respect and has historical roots.

Here's a quote from the FAQ:

Why does Python use methods for some functionality (e.g. list.index()) but functions for other (e.g. len(list))?

The major reason is history. Functions were used for those operations that were generic for a group of types and which were intended to work even for objects that didn’t have methods at all (e.g. tuples). It is also convenient to have a function that can readily be applied to an amorphous collection of objects when you use the functional features of Python (map(), apply() et al).

In fact, implementing len(), max(), min() as a built-in function is actually less code than implementing them as methods for each type. One can quibble about individual cases but it’s a part of Python, and it’s too late to make such fundamental changes now. The functions have to remain to avoid massive code breakage.

The other "magical methods" (actually called special method in the Python folklore) make lots of sense, and similar functionality exists in other languages. They're mostly used for code that gets called implicitly when special syntax is used.

For example:

• overloaded operators (exist in C++ and others)
• constructor/destructor
• hooks for accessing attributes
• tools for metaprogramming

and so on...

Solution 2

From the Zen of Python:

In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.

This is one of the reasons - with custom methods, developers would be free to choose a different method name, like getLength(), length(), getlength() or whatsoever. Python enforces strict naming so that the common function len() can be used.

All operations that are common for many types of objects are put into magic methods, like __nonzero__, __len__ or __repr__. They are mostly optional, though.

Operator overloading is also done with magic methods (e.g. __le__), so it makes sense to use them for other common operations, too.

Solution 3

Python uses the word "magic methods", because those methods really performs magic for you program. One of the biggest advantages of using Python's magic methods is that they provide a simple way to make objects behave like built-in types. That means you can avoid ugly, counter-intuitive, and nonstandard ways of performing basic operators.

Consider a following example:

dict1 = {1 : "ABC"}
dict2 = {2 : "EFG"}

dict1 + dict2
Traceback (most recent call last):
  File "python", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'dict' and 'dict'

This gives an error, because the dictionary type doesn't support addition. Now, let's extend dictionary class and add "__add__" magic method:

class AddableDict(dict):

    def __add__(self, otherObj):
        self.update(otherObj)
        return AddableDict(self)


dict1 = AddableDict({1 : "ABC"})
dict2 = AddableDict({2 : "EFG"})

print (dict1 + dict2)

Now, it gives following output.

{1: 'ABC', 2: 'EFG'}

Thus, by adding this method, suddenly magic has happened and the error you were getting earlier, has gone away.

I hope, it makes things clear to you. For more information, refer to:

A Guide to Python's Magic Methods (Rafe Kettler, 2012)

Solution 4

Some of these functions do more than a single method would be able to implement (without abstract methods on a superclass). For instance bool() acts kind of like this:

def bool(obj):
    if hasattr(obj, '__nonzero__'):
        return bool(obj.__nonzero__())
    elif hasattr(obj, '__len__'):
        if obj.__len__():
            return True
        else:
            return False
    return True

You can also be 100% sure that bool() will always return True or False; if you relied on a method you couldn't be entirely sure what you'd get back.

Some other functions that have relatively complicated implementations (more complicated than the underlying magic methods are likely to be) are iter() and cmp(), and all the attribute methods (getattr, setattr and delattr). Things like int also access magic methods when doing coercion (you can implement __int__), but do double duty as types. len(obj) is actually the one case where I don't believe it's ever different from obj.__len__().

Solution 5

They are not really "magic names". It's just the interface an object has to implement to provide a given service. In this sense, they are not more magic than any predefined interface definition you have to reimplement.

View more solutions

Comments

[Grant Paul]

I think the general reason is a) historical and b) something like len() or reversed() applies to many types of objects, but a method such as append() only applies to sequences, etc.

[visual_learner]

Instead of hasattr() I would use try: / except AttributeError: and instead of the if obj.__len__(): return True else: return False I would just say return obj.__len__() > 0 but those are just stylistic things.

[Ponkadoodle]

In python 2.6 (which btw bool(x) referred to x.__nonzero__()), your method wouldn't work. bool instances have a method __nonzero__(), and your code would keep calling itself once obj was a bool. Perhaps bool(obj.__bool__()) should be treated the same way you treated __len__? (Or does this code actually work for Python 3?)

[Ian Bicking]

The circular nature of bool() was somewhat intentionally absurd, to reflect the peculiarly circular nature of the definition. There's an argument that it should simply be considered a primitive.

[ncoghlan]

The only difference (currently) between len(x) and x.__len__() is that the former will raise OverflowError for lengths that exceed sys.maxsize, while the latter generally won't for types implemented in Python. That's more a bug than a feature, though (e.g. Python 3.2's range object can mostly handle arbitrarily large ranges, but using len with them may fail. Their __len__ fails as well though, since they're implemented in C rather than Python)

[jpmc26]

Python and the Principle of Least Astonishment is a good read for some of the advantages to Python being this way (although I do admit the English needs work). The basic point: it allows the standard library to implement a ton of code that becomes very, very reusable but still overridable.

[Andy Hayden]

This is a compelling argument. More satisfying that "Guido didn't really believe in OO".... (as I have seen claimed elsewhere).