I don't understand this python __del__ behaviour

Question

Can someone explain why the following code behaves the way it does:

import types

class Dummy():
    def __init__(self, name):
        self.name = name
    def __del__(self):
        print "delete",self.name

d1 = Dummy("d1")
del d1
d1 = None
print "after d1"

d2 = Dummy("d2")
def func(self):
    print "func called"
d2.func = types.MethodType(func, d2)
d2.func()
del d2
d2 = None
print "after d2"

d3 = Dummy("d3")
def func(self):
    print "func called"
d3.func = types.MethodType(func, d3)
d3.func()
d3.func = None
del d3
d3 = None
print "after d3"

The output (note that the destructor for d2 is never called) is this (python 2.7)

delete d1
after d1
func called
after d2
func called
delete d3
after d3

Is there a way to "fix" the code so the destructor is called without deleting the method added? I mean, the best place to put the d2.func = None would be in the destructor!

Thanks

[edit] Based on the first few answers, I'd like to clarify that I'm not asking about the merits (or lack thereof) of using __del__. I tried to create the shortest function that would demonstrate what I consider to be non-intuitive behavior. I'm assuming a circular reference has been created, but I'm not sure why. If possible, I'd like to know how to avoid the circular reference....

Solution 1

I'm providing my own answer because, while I appreciate the advice to avoid __del__, my question was how to get it to work properly for the code sample provided.

Short version: The following code uses weakref to avoid the circular reference. I thought I'd tried this before posting the question, but I guess I must have done something wrong.

import types, weakref

class Dummy():
    def __init__(self, name):
        self.name = name
    def __del__(self):
        print "delete",self.name

d2 = Dummy("d2")
def func(self):
    print "func called"
d2.func = types.MethodType(func, weakref.ref(d2)) #This works
#d2.func = func.__get__(weakref.ref(d2), Dummy) #This works too
d2.func()
del d2
d2 = None
print "after d2"

Longer version: When I posted the question, I did search for similar questions. I know you can use with instead, and that the prevailing sentiment is that __del__ is BAD.

Using with makes sense, but only in certain situations. Opening a file, reading it, and closing it is a good example where with is a perfectly good solution. You've gone a specific block of code where the object is needed, and you want to clean up the object and the end of the block.

A database connection seems to be used often as an example that doesn't work well using with, since you usually need to leave the section of code that creates the connection and have the connection closed in a more event-driven (rather than sequential) timeframe.

If with is not the right solution, I see two alternatives:

1. You make sure __del__ works (see this blog for a better description of weakref usage)
2. You use the atexit module to run a callback when your program closes. See this topic for example.

While I tried to provide simplified code, my real problem is more event-driven, so with is not an appropriate solution (with is fine for the simplified code). I also wanted to avoid atexit, as my program can be long-running, and I want to be able to perform the cleanup as soon as possible.

So, in this specific case, I find it to be the best solution to use weakref and prevent circular references that would prevent __del__ from working.

This may be an exception to the rule, but there are use-cases where using weakref and __del__ is the right implementation, IMHO.

Solution 2

You cannot assume that __del__ will ever be called - it is not a place to hope that resources are automagically deallocated. If you want to make sure that a (non-memory) resource is released, you should make a release() or similar method and then call that explicitly (or use it in a context manager as pointed out by Thanatos in comments below).

At the very least you should read the __del__ documentation very closely, and then you should probably not try to use __del__. (Also refer to the gc.garbage documentation for other bad things about __del__)

Solution 3

Instead of del, you can use the with operator.

http://effbot.org/zone/python-with-statement.htm

just like with filetype objects, you could something like

with Dummy('d1') as d:
    #stuff
#d's __exit__ method is guaranteed to have been called

Solution 4

del doesn't call __del__

del in the way you are using removes a local variable. __del__ is called when the object is destroyed. Python as a language makes no guarantees as to when it will destroy an object.

CPython as the most common implementation of Python, uses reference counting. As a result del will often work as you expect. However it will not work in the case that you have a reference cycle.

d3 -> d3.func -> d3

Python doesn't detect this and so won't clean it up right away. And its not just reference cycles. If an exception is throw you probably want to still call your destructor. However, Python will typically hold onto to the local variables as part of its traceback.

The solution is not to depend on the __del__ method. Rather, use a context manager.

class Dummy:
   def __enter__(self):
       return self

   def __exit__(self, type, value, traceback):
       print "Destroying", self

with Dummy() as dummy:
    # Do whatever you want with dummy in here
# __exit__ will be called before you get here

This is guaranteed to work, and you can even check the parameters to see whether you are handling an exception and do something different in that case.

Solution 5

A full example of a context manager.

class Dummy(object):
    def __init__(self, name):
        self.name = name
    def __enter__(self):
        return self
    def __exit__(self, exct_type, exce_value, traceback):
        print 'cleanup:', d
    def __repr__(self):
        return 'Dummy(%r)' % (self.name,)

with Dummy("foo") as d:
    print 'using:', d

print 'later:', d

View more solutions

Comments

[Rafe Kettler]

Your code for __del__ doesn't actually delete anything. Regardless, using __del__ is not safe.

[Jochen Ritzel]

Yeah, types.MethodType(func, d2) has a reference to d2 and you put that on d2, so you have a circular reference. Nothing surprising about it, but why do you do that if that's not what you want?

[Brett Stottlemyer]

@Jochen Ritzel - I DO want a dynamically bound method. I've created a class that adds methods (getattr) as needed. This is a long running class, so using with isn't a good option, so __ del__ seems like a great place to clean up. I was hoping there was a simple way to remove the circular dependency. The code above is simplified to show the issue. I think I'm inclined to go with an atexit solution.

[Brett Stottlemyer]

Here's the link to a similar question and an atexit answer: link

[user2471801]

"Is there a way to "fix" the code so the destructor is called without deleting the method added?" I think we're all confused by this statement. If your deleting an instance of a specific object, that you've added a specific method to, why do you need that method to continue to exist? If you want it to be a part of the Dummy object then you can add the method to it: Dummy.func = types.MethodType(func, Dummy). This will then add the method to all instances of Dummy. (Which may not be the desired behavior, but that's what you'd get)

[Brett Stottlemyer]

@monkut - sorry for the confusion. d2/d3 show that the destructor is called "as expected" only if the added function is first removed from the instance. My goal is to have the class destructor called when the object goes out of scope, without my having to call any cleanup function manually. d3 isn't feasible because it requires a cleanup function before __ del__ runs, which doesn't make sense. I'm wondering if there is some sort of fix (like weakref?) that will make d2's destructor get called. Make more sense?

[Jochen Ritzel]

@Brett: You're doing something weird (adding functions to instances) and it's causing problems, so you're looking for more weird things to fix that. It would be much more reasonable to take a step back and ask for a way to solve your original problem in a way that doesn't cause more problems. You could fix this problem with descriptors, but I'm hesitant to explain how - it is kind of complicated and the only thing worse than a complicated solution is a complicated solution to a avoidable problem.

[Brett Stottlemyer]

"You could fix this problem with descriptors, but I'm hesitant to explain how". Didn't expect to see that type of comment on Stackoverflow. I guess I will have to look up descriptors myself.

[Aryeh Leib Taurog]

@JochenRitzel Descriptors won't help here, since all the descriptor would do is instantiate the MethodType object behind the scenes. This would still create a circular reference.

[Rafe Kettler]

Exactly right. __del__ is usually a bad idea. Being explicit is always a good idea.

[Thanatos]

Additionally, the with statement can be used in conjunction with a release()-like function to write simpler, cleaner code.

[SingleNegationElimination]

d is not out of scope. the only guarantee is that the __exit__ method was called on object created by Dummy('d1')

[SingleNegationElimination]

It's fantastically important to remember that there is just no guarantee on whether __del__ will be called, when it gets called, or how many times it gets called. There are all sorts of ways which, out of your control, garbage collection happens who knows when. In particular, cPython is the only python that does reference counting and eagerly collects such garbage. Jython, PyPy (and IronPython, I think) collect garbage only during sweeps. Depending on __del__ is the surest path to disaster.

[Falmarri]

@TokenMacGuy: ok, you're right. It's not technically out of scope, I guess that's worth mentioning. However, to use the syntax, the object MUST implement then __enter__ and __exit__ methods. Hence, for all intents and purposes, it goes out of scope after the with statement, as one would presumably write their __exit__ method as if it were a destructor. I didn't, however, realize that it didn't ACTUALLY go out of scope, which I imagine can be useful for all sorts of hacking

[Nick Bastin]

Just to follow up on TokenMacGuy's comment - the null garbage collector is a perfectly valid implementation if you never run out of memory, and it will result in __del__ never being called. GC's are not aware of non-memory resources, and shouldn't be relied upon to manage them.

[Brett Stottlemyer]

@Nick - GC's are not aware of non-memory resources? Of course they are. They are aware of anything they count references from. If you think of the reference as memory (in which case every object is memory and works with the GC) your statement is meaningless. If you don't count the reference as memory, your statement is incorrect.

[Nick Bastin]

@BrettStottlemyer: No, the Python GC is not specifically aware of any non-memory resource - the only thing it knows how to do is find loose objects and delete them (from memory). Because Python doesn't have destructors that the GC will trigger, this means the GC has no side effects, and won't close database connections, etc. Yes, yes, of course you could write a database connection GC, but the fact is that Python doesn't have one (in any implementation), and the Python GC implementations merely clean up the only resource they know about - memory.

[Ethan Furman]

weakref is the right solution to the problem you presented; I'm curious, though, why you don't add the method to the class instead of the instance? That would avoid the problem all together.

[Brett Stottlemyer]

@EthanFurman - Because, in my case, adding the functions is dynamic (at runtime) and not known in advance.

[Ethan Furman]

@BrettStottlemyer: Sorry, I phrased that poorly. It is possible to add to the class just as easily as to the instance; the question is whether all instances of that class need the additional methods. Also, if you assign to the class instead of the instance you don't have to mess with MethodType or circular references -- it's just Dummy.func = func, or d2.__class__.func = func.

[Tom Swirly]

If "being explicit" is always a good idea, why have garbage collection at all? "Remembering to call" a release function is fragile. You might forget to call it - or an exception might be raised at an unexpected point and you might never reach the release function. This article explains how you can guarantee that your del function is called. If possible, a context manager is even better, but much of the time it isn't possible, as your object's lifespan transcends one method or function.

[Tom Swirly]

"It's fantastically important to remember that there is just no guarantee on whether del will be called". There are strict, explicit conditions on whether del will be called - please see this reference.

[Nick Bastin]

@TomSwirly: You may be able to guarantee your object will qualify for garbage collection, but you cannot guarantee when it will be collected. If the garbage collector is free to collect an object, it is also free to not collect it. This blog post from Raymond Chen is always worth remembering. Assuming too much about a given GC implementation might take today's working code and mysteriously break it tomorrow.

[Aryeh Leib Taurog]

Explicit is better than implicit. If you need to open a connection in one event handler and close it in another, call close() explicitly! If not, wrap your whole event loop in a with statement. Use contextlib.closing if need be. If you don't know where to close your connections, then you've a much bigger problem. Instead of hacking at the language so your sloppy magic code will work, figure out where to put those close() statements. You'll be glad you did.

[Aryeh Leib Taurog]

-1. You have to know how to ask a question, or at least how to take an answer. See this meta question and this one on the XY Problem. You asked "Why doesn't __del__ seem to work," which is a legitimate question about python internals, to which you got a legitimate answer. Your real problem was "How do I close a resource in one event handler that I've opened in another," the answer to which has nothing to do with __del__.

[Brett Stottlemyer]

@Aryeh - I disagree. My question was how to get a feature of python to work, and I did provide a correct answer. If the real answer is to not use __ del__, then __ del__ should be removed from python. I believe that the upvotes give people an idea of what is commonly accepted, which my answer isn't. That's fine. But how can you say my answer is wrong, when it DOES answer the question (as asked, not as you incorrectly restated it)?

[jpmc26]

@TomSwirly If you can't identify a single scope in which a particular resource is held and then released, then you should reconsider how your code is organized. Creating a single scope where you can acquire and release a resource is the best plan, even when it requires significant refactoring. You can then pass your resource object into a function call or what have you and keep passing it down the stack as needed. Often, you can find a way to hold a resource for only a short section of the code (generate content before opening a file for writing, close connection as soon as data is read).

[jpmc26]

Regarding a database connection as an example that doesn't work well using with, I strongly disagree. It is generally very simple to acquire a database at some high scope of your program, pass it as an argument into other portions of code, and then release it once they return. It might be executing a lot of code between passing the argument and returning, but it's still a very clearly defined scope. Favor those kinds of code over using globals for database connections, to solve exactly this sort of problem.

[Nick Bastin]

@jpmc26: That is certainly good advice, although what I say above still applies - qualifying for garbage collection is not the same thing as guaranteeing that you will be collected. If your resource isn't memory (the only thing the garbage collector is designed to do reasonable things with), you need to take a more proactive role (like using a context manager, which is within the scope (no pun intended.. :-)) of what you are suggesting).

[jpmc26]

@NickBastin My apologies for not being clear enough. I was trying to say that when you find yourself "unable" to use a context manager, reorganize your code so you can use one. =) I thought my mentions of "acquire" and "release" would be understood as the functions a context manager typically performs. I agree with you 110%.

[DylanYoung]

There are, of course, a million other ways to get similar functionality without the circular reference.

[Ainz Titor]

@BrettStottlemyer I'm very grateful that you answered your own question, while others complained about del without a helpful explanation. I found your solution to be very illuminating about del's functionality. I upvoted both your question and answer without hesitation.

[antred]

In my opinion, since Python has context managers and __del is essentially useless it should have been removed in Python 3.

[D. A.]

It might be helpful to have specific details on the precise problem your example demonstrates. I found it here: eli.thegreenplace.net/2009/06/12/… It refers to this documentation: docs.python.org/2/reference/datamodel.html Which states: "Circular references which are garbage are detected when the option cycle detector is enabled (it’s on by default), but can only be cleaned up if there are no Python-level __del__() methods involved"

[Hejazzman]

@AinzTitor If so, then you upvoted a misguided answer that taught you the wrong way to write the code.