python list __iter__ method called on every loop?

Question

I am trying to make a class which inherits from a python list. I want the elements of the list to be initialized/finalized with every loop of the list. I thought this could be done by overriding the __iter__ method of the python list but I can't seem to get it to work. The __iter__ method appears to called only once ? (see below)

class MyList(list):
    def __iter__(self):
        print 'do something'
        return list.__iter__(self)

my_list = MyList(range(10))
print my_list

for item in my_list:
    print item

Output

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
do something
0
1
2
3
4
5
6
7
8
9

Any idea how I can achieve what I want to do?

Solution 1

You can simply return a generator expression from __iter__()

class MyList(list):
    def __iter__(self):
        return (self.do_something(x) for x in list.__iter__(self))

    def do_something(self, x):
        print 'do something', x
        return x

my_list = MyList(range(10))
print my_list

for item in my_list:
    print item

ncoghlan suggests using a generator in place of the generator expression which makes for easier debugging

class MyList(list):

    def __iter__(self):
        for x in list.__iter__(self):
            yield self.do_something(x)

    def do_something(self, x):
        print 'do something', x
        return x

my_list = MyList(range(10))
print my_list

for item in my_list:
    print item

alternatively you could use imap here

from itertools import imap

class MyList(list):
    def __iter__(self):
        return imap(self.do_something, list.__iter__(self))

    def do_something(self, x):
        print 'do something', x
        return x


my_list = MyList(range(10))
print my_list

for item in my_list:
    print item

Solution 2

__iter__ returns an iterator object. If you need to do something in every iteration, you have to implement your own (it has to implement two methods described in the linked docs).

Solution 3

The python class __iter__() method actually returns an iterator object. See the following for reference: http://docs.python.org/library/stdtypes.html#iterator-types

On the iterator object the next() method will be called on each step of the loop. You could write a custom iterator which would be returned by your custom list.

Comments

[ncoghlan]

I was going to vote this up, but why return a generator expression instead of just implementing a generator?

[leos]

would you please be more specific?

[Winston Ewert]

next will never be called on the list object. You have to define an iter method which returns an iterator object. The next method will be called on that returned iterator object.

[John La Rooy]

@Winston, I'm not calling obj.__iter__, i'm calling list.__iter__ it's necessary to do it that way to avoid endless recursion

[John La Rooy]

@ncoghlan, why would you implement a generator where a GE can suffice?

[ncoghlan]

You get much better debugging information when a generator function goes wrong (since it knows its own name). A GE is useful if you specifically need an expression, but if you're defining a named function anyway, then it is better to actually make that a generator function: def __iter__(self): for x in list.__iter__(self): yield self.do_something(x) (oh, for vertical whitespace in comments...)

[John La Rooy]

@ncoghlan, OK I've added it to my answer so you don't have to lose sleep about whitespace in comments

[ncoghlan]

Thanks :) It may also be an artefact of already knowing Python before GEs were added. For a very long time writing a generator function was simply the way to write __iter__ methods if there wasn't an existing iterator with the correct behaviour.