Python List & for-each access (Find/Replace in built-in list)

Question

I originally thought Python was a pure pass-by-reference language.

Coming from C/C++ I can't help but think about memory management, and it's hard to put it out of my head. So I'm trying to think of it from a Java perspective and think of everything but primitives as a pass by reference.

Problem: I have a list, containing a bunch of instances of a user defined class.

If I use the for-each syntax, ie.:

for member in my_list:
    print(member.str);

Is member the equivalent of an actual reference to the object?

Is it the equivalent of doing:

i = 0
while i < len(my_list):
    print(my_list[i])
    i += 1

I think it's NOT, because when I'm looking to do a replace, it doesn't work, that is, this doesn't work:

for member in my_list:
    if member == some_other_obj:
        member = some_other_obj

A simple find and replace in a list. Can that be done in a for-each loop, if so, how? Else, do I simply have to use the random access syntax (square brackets), or will NEITHER work and I need to remove the entry, and insert a new one? I.e.:

i = 0
for member in my_list:
   if member == some_other_obj:
      my_list.remove(i)
      my_list.insert(i, member)
   i += 1

Solution 1

Answering this has been good, as the comments have led to an improvement in my own understanding of Python variables.

As noted in the comments, when you loop over a list with something like for member in my_list the member variable is bound to each successive list element. However, re-assigning that variable within the loop doesn't directly affect the list itself. For example, this code won't change the list:

my_list = [1,2,3]
for member in my_list:
    member = 42
print my_list

Output:

[1, 2, 3]

If you want to change a list containing immutable types, you need to do something like:

my_list = [1,2,3]
for ndx, member in enumerate(my_list):
    my_list[ndx] += 42
print my_list

Output:

[43, 44, 45]

If your list contains mutable objects, you can modify the current member object directly:

class C:
    def __init__(self, n):
        self.num = n
    def __repr__(self):
        return str(self.num)

my_list = [C(i) for i in xrange(3)]
for member in my_list:
    member.num += 42
print my_list

[42, 43, 44]

Note that you are still not changing the list, simply modifying the objects in the list.

You might benefit from reading Naming and Binding.

Solution 2

Python is not Java, nor C/C++ -- you need to stop thinking that way to really utilize the power of Python.

Python does not have pass-by-value, nor pass-by-reference, but instead uses pass-by-name (or pass-by-object) -- in other words, nearly everything is bound to a name that you can then use (the two obvious exceptions being tuple- and list-indexing).

When you do spam = "green", you have bound the name spam to the string object "green"; if you then do eggs = spam you have not copied anything, you have not made reference pointers; you have simply bound another name, eggs, to the same object ("green" in this case). If you then bind spam to something else (spam = 3.14159) eggs will still be bound to "green".

When a for-loop executes, it takes the name you give it, and binds it in turn to each object in the iterable while running the loop; when you call a function, it takes the names in the function header and binds them to the arguments passed; reassigning a name is actually rebinding a name (it can take a while to absorb this -- it did for me, anyway).

With for-loops utilizing lists, there are two basic ways to assign back to the list:

for i, item in enumerate(some_list):
    some_list[i] = process(item)

or

new_list = []
for item in some_list:
    new_list.append(process(item))
some_list[:] = new_list

Notice the [:] on that last some_list -- it is causing a mutation of some_list's elements (setting the entire thing to new_list's elements) instead of rebinding the name some_list to new_list. Is this important? It depends! If you have other names besides some_list bound to the same list object, and you want them to see the updates, then you need to use the slicing method; if you don't, or if you do not want them to see the updates, then rebind -- some_list = new_list.

Solution 3

You could replace something in there by getting the index along with the item.

>>> foo = ['a', 'b', 'c', 'A', 'B', 'C']
>>> for index, item in enumerate(foo):
...     print(index, item)
...
(0, 'a')
(1, 'b')
(2, 'c')
(3, 'A')
(4, 'B')
(5, 'C')
>>> for index, item in enumerate(foo):
...     if item in ('a', 'A'):
...         foo[index] = 'replaced!'
...
>>> foo
['replaced!', 'b', 'c', 'replaced!', 'B', 'C']

Note that if you want to remove something from the list you have to iterate over a copy of the list, else you will get errors since you're trying to change the size of something you are iterating over. This can be done quite easily with slices.

Wrong:

>>> foo = ['a', 'b', 'c', 1, 2, 3]
>>> for item in foo:
...     if isinstance(item, int):
...         foo.remove(item)
...
>>> foo 
['a', 'b', 'c', 2]

The 2 is still in there because we modified the size of the list as we iterated over it. The correct way would be:

>>> foo = ['a', 'b', 'c', 1, 2, 3]
>>> for item in foo[:]:
...     if isinstance(item, int):
...         foo.remove(item)
...
>>> foo 
['a', 'b', 'c']

Comments

[Syndacate]

How unfortunate that it's re-defined as a copy of the list member. I believe it would be much more useful to show up as a reference, as if you were setting it, it would probably be to manipulate the structure in some way or another. I tested the syntax: for idx in range(0, len(my_list)): my_list[idx] = new_obj It works to my liking. Thank you.

[jfs]

@neurino: pythonic approach would be: foo = [c for c in foo if condition(c)]

[GreenMatt]

@Syndacate: Not sure how C++ handles things like this, but Java's version of the for each loop (or enhanced for loop in their lingo) works in a similar fashion, as least in effect.

[GreenMatt]

@J.F. Sebastian: I may not have the terminology perfect, but I believe I've provided a good way to think about this. The answer has been changed & expanded.

[neurino]

@J.F. Sebastian: honestly I don't think everything that can fit in a one-liner is more pythonic than 3 lines of code, I wanted to emphasize that Gilder was using enumerate while GreenMat was stuck (he edited his answer) to for x in xrange(len()). Cheers

[GreenMatt]

@neurino: Granted I'm an "old school" programmer trying to update my skills myself; thus I tend to think for x in xrange(len(l)) first. As such, I expect for x in xrange(len(l)) will be more familiar than for x in enumerate(l) to someone coming from C/C++ (which the OP professed to being). Is it better to be more Pythonic or to focus on the core issue and use more familiar syntax on things that are necessary for the example but not central to the problem? (I don't think there's a single correct answer to that.)

[jfs]

@neurino: I've used list comprehension to replace O(N**2) loop over foo[:] with foo.remove() in it (3rd example). I've nothing against the first example with enumerate(). It should be obvious due to 1st and 3rd examples do different things and the list comprehension produces similar result to the 3rd example.

[jfs]

@GreenMatt: I can't revoke my vote unless you edit the answer (it is a feature of SO).

[jfs]

I remember the whole "Names refer to objects. Names are introduced by name binding operations" terminology just clicked when I saw python.net/~goodger/projects/pycon/2007/idiomatic/… So you don't change change member you just bind it to another object. And though member is not one of the objects in the list at any moment, it refers to each of them.

[neurino]

@GreenMatt: I agree being pythonic is not mandatory however if I never heard about being javanic or *C*inic is because python offers a slight different way of thinking about programming and writing code, the early you get it the better IMHO. Anyway there are situations where all you need is and index and nothing else, in that case I go with xrange(len(l)) and don't feel being less pythonic... :)

[neurino]

@J.F. Sebastian: honestly in the same situation (3rd example) I'd go with list comprehension too (if condition does not appear to be cumbersome or awkward) Cheers

[GreenMatt]

@J.F. Sebastian: I think I had edited the entry after your downvote. However, I have edited it again (hopefully improving it).

[jfs]

btw, there are no primitives in Python unlike in Java. Integers are (immutable) objects.

[Syndacate]

@neurino: I have little interest in it being pythonic, the code is for me for a class, if it was up to me it'd be in C++, but it's not up to me. It needs to work correctly, that is all. Every language has their "nics" - C gurus try to be very tricky with some utility and syntax black magic, Java gurus try to design a hello world program to be modular, haskell gurus pride themselves in 1 line recursively functional function calls, etc. I don't think there's any really "correct" answer, either, just what one prefers. That being said, I do now understand the name-binding concept python uses.

[neurino]

@Syndacate: you should have written you have little interest in python at all otherwise you should care about language specs and zen. Cheers

[r.v]

The name binding works the same way as in Java. I could not see any difference.

[Sqeaky]

In java primitives like ints and booleans are passed by value and everything not a primitive is passed by reference. Binding to a name is similar to passing by reference, minus the type safety and probably a few other details. In java a reference maintains, at least into compilation, type data about what it can refer to python names have no such restriction.