Python (and Python C API): __new__ versus __init__

Question

The question I'm about to ask seems to be a duplicate of Python's use of __new__ and __init__?, but regardless, it's still unclear to me exactly what the practical difference between __new__ and __init__ is.

Before you rush to tell me that __new__ is for creating objects and __init__ is for initializing objects, let me be clear: I get that. In fact, that distinction is quite natural to me, since I have experience in C++ where we have placement new, which similarly separates object allocation from initialization.

The Python C API tutorial explains it like this:

The new member is responsible for creating (as opposed to initializing) objects of the type. It is exposed in Python as the __new__() method. ... One reason to implement a new method is to assure the initial values of instance variables.

So, yeah - I get what __new__ does, but despite this, I still don't understand why it's useful in Python. The example given says that __new__ might be useful if you want to "assure the initial values of instance variables". Well, isn't that exactly what __init__ will do?

In the C API tutorial, an example is shown where a new Type (called a "Noddy") is created, and the Type's __new__ function is defined. The Noddy type contains a string member called first, and this string member is initialized to an empty string like so:

static PyObject * Noddy_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    .....

    self->first = PyString_FromString("");
    if (self->first == NULL)
    {
       Py_DECREF(self);
       return NULL;
    }

    .....
}

Note that without the __new__ method defined here, we'd have to use PyType_GenericNew, which simply initializes all of the instance variable members to NULL. So the only benefit of the __new__ method is that the instance variable will start out as an empty string, as opposed to NULL. But why is this ever useful, since if we cared about making sure our instance variables are initialized to some default value, we could have just done that in the __init__ method?

Solution 1

The difference mainly arises with mutable vs immutable types.

__new__ accepts a type as the first argument, and (usually) returns a new instance of that type. Thus it is suitable for use with both mutable and immutable types.

__init__ accepts an instance as the first argument and modifies the attributes of that instance. This is inappropriate for an immutable type, as it would allow them to be modified after creation by calling obj.__init__(*args).

Compare the behaviour of tuple and list:

>>> x = (1, 2)
>>> x
(1, 2)
>>> x.__init__([3, 4])
>>> x # tuple.__init__ does nothing
(1, 2)
>>> y = [1, 2]
>>> y
[1, 2]
>>> y.__init__([3, 4])
>>> y # list.__init__ reinitialises the object
[3, 4]

As to why they're separate (aside from simple historical reasons): __new__ methods require a bunch of boilerplate to get right (the initial object creation, and then remembering to return the object at the end). __init__ methods, by contrast, are dead simple, since you just set whatever attributes you need to set.

Aside from __init__ methods being easier to write, and the mutable vs immutable distinction noted above, the separation can also be exploited to make calling the parent class __init__ in subclasses optional by setting up any absolutely required instance invariants in __new__. This is generally a dubious practice though - it's usually clearer to just call the parent class __init__ methods as necessary.

Solution 2

__new__() can return objects of types other than the class it's bound to. __init__() only initializes an existing instance of the class.

>>> class C(object):
...   def __new__(cls):
...     return 5
...
>>> c = C()
>>> print type(c)
<type 'int'>
>>> print c
5

Solution 3

There are probably other uses for __new__ but there's one really obvious one: You can't subclass an immutable type without using __new__. So for example, say you wanted to create a subclass of tuple that can contain only integral values between 0 and size.

class ModularTuple(tuple):
    def __new__(cls, tup, size=100):
        tup = (int(x) % size for x in tup)
        return super(ModularTuple, cls).__new__(cls, tup)

You simply can't do this with __init__ -- if you tried to modify self in __init__, the interpreter would complain that you're trying to modify an immutable object.

Solution 4

Not a complete answer but perhaps something that illustrates the difference.

__new__ will always get called when an object has to be created. There are some situations where __init__ will not get called. One example is when you unpickle objects from a pickle file, they will get allocated (__new__) but not initialised (__init__).

Solution 5

Just want to add a word about the intent (as opposed to the behavior) of defining __new__ versus __init__.

I came across this question (among others) when I was trying to understand the best way to define a class factory. I realized that one of the ways in which __new__ is conceptually different from __init__ is the fact that the benefit of __new__ is exactly what was stated in the question:

So the only benefit of the __new__ method is that the instance variable will start out as an empty string, as opposed to NULL. But why is this ever useful, since if we cared about making sure our instance variables are initialized to some default value, we could have just done that in the __init__ method?

Considering the stated scenario, we care about the initial values of the instance variables when the instance is in reality a class itself. So, if we are dynamically creating a class object at runtime and we need to define/control something special about the subsequent instances of this class being created, we would define these conditions/properties in a __new__ method of a metaclass.

I was confused about this until I actually thought about the application of the concept rather than just the meaning of it. Here's an example that would hopefully make the difference clear:

a = Shape(sides=3, base=2, height=12)
b = Shape(sides=4, length=2)
print(a.area())
print(b.area())

# I want `a` and `b` to be an instances of either of 'Square' or 'Triangle'
# depending on number of sides and also the `.area()` method to do the right
# thing. How do I do that without creating a Shape class with all the
# methods having a bunch of `if`s ? Here is one possibility

class Shape:
    def __new__(cls, sides, *args, **kwargs):
        if sides == 3:
            return Triangle(*args, **kwargs)
        else:
            return Square(*args, **kwargs)

class Triangle:
    def __init__(self, base, height):
        self.base = base
        self.height = height

    def area(self):
        return (self.base * self.height) / 2

class Square:
    def __init__(self, length):
        self.length = length

    def area(self):
        return self.length*self.length

Note this is just an demonstartive example. There are multiple ways to get a solution without resorting to a class factory approach like above and even if we do choose to implelent the solution in this manner, there are a little caveats left out for sake of brevity (for instance, declaring the metaclass explicitly)

If you are creating a regular class (a.k.a a non-metaclass), then __new__ doesn't really make sense unless it is special case like the mutable versus immutable scenario in ncoghlan's answer answer (which is essentially a more specific example of the concept of defining the initial values/properties of the class/type being created via __new__ to be then initialized via __init__).

View more solutions

Comments

[Alcott]

I don't understand why should we use super? I mean why should new return an instance of the superclass? Furthermore, as your put it, why should we pass cls explicitly to new? super(ModularTuple, cls) doesn't return a bound method?

[senderle]

@Alcott, I think you're misunderstanding the behavior of __new__. We pass cls explicitly to __new__ because, as you can read here __new__ always requires a type as its first argument. It then returns an instance of that type. So we aren't returning an instance of the superclass -- we're returning an instance of cls. In this case, it's just the same as if we had said tuple.__new__(ModularTuple, tup).

[Tarik]

This is the leanest explanation so far.

[Miles Rout]

the code you refer to as "boilerplate" in __new__ isn't boilerplate, because boilerplate never changes. Sometimes you need to replace that particular code with something different.

[ncoghlan]

Creating, or otherwise acquiring, the instance (usually with a super call) and returning the instance are necessary parts of any __new__ implementation, and the "boilerplate" I am referring to. By contrast, pass is a valid implementation for __init__ - there is no required behaviour whatsoever.

[redpix_]

Would I call init from new if I wanted memory to be allocated and data to be initialized? Why if new doesn't exist when creating instance init gets called?

[Noufal Ibrahim]

The job of the __new__ method is to create (this implies memory allocation) an instance of the class and return it. The initialisation is a separate step and it's what is usually visible to the user. Please ask a separate question if there's a specific problem you're facing.

[ArtOfWarfare]

Not quite true. I have __init__ methods which contain code that looks like self.__class__ = type(...). That causes the object to be of a different class than the one you thought you were creating. I can't actually change it into an int as you did... I get an error about heap types or something... but my example of assigning it to a dynamically created class works.

[martineau]

I, too, am confused about when __init__() is called. For example, in lonetwin's answer, either Triangle.__init__() or Square.__init__() automatically get called depending on which type __new__() returns. From what you say in your answer (and I've read this elsewhere), it doesn't seem like either of them should be since Shape.__new__() isn't returning an instance of cls (nor one of a subclass of it).

[Ignacio Vazquez-Abrams]

@martineau: The __init__() methods in lonetwin's answer get called when the individual objects get instantiated (i.e. when their __new__() method returns), not when Shape.__new__() returns.

[martineau]

Ahh, right, Shape.__init__() (if it had one) wouldn't be called. Now it's all making more sense... :¬)

[sakeesh]

@senderle. can we say that the satement super(ModularTuple, cls).__new__(cls, tup) is same as super().__new__(cls, tup)

[senderle]

@sakeesh ah, this is a very old answer, and does not account for the Python 3 way of doing things. In Python 3, I think you are right, but I will have to look up the details to be sure.

[vicmortelmans]

Just one caveat to the above: initialization should be done inside the __new__ function. If you have a separate __init__ function, it will be executed every time the constructor is called, which is probably not what you want.

[pyansharp]

Is this supposed to result in the parent class' __new__() getting called twice?

[pyansharp]

nvm, I didn't realize the classes in this example are not inheriting from Shape.

[Maggyero]

This is the clearest answer on the topic. Extra remark for readers: tuple.__init__ does nothing because it resolves to object.__init__ (which does nothing), whereas list.__init__ (which initialises the instance) overrides object.__init__.

[Algo]

Excellent explaination.