Appending a numpy array to a list - strange happenings

Question

Using Python3.5.3 in Spyder 3.1.3 on Raspbian on a Raspberry Pi. Appending two numpy-arrays to a list named 'list0' works well with allocated numpy arrays 'a' like:

import numpy as np

list0 = []
a = np.array([[1,2,3],[2,3,4]])
list0.append(a)
a = np.array([[11,12,13],[12,13,14]])
list0.append(a)

print("list0 =",list0)

works well, giving as output (a bit better formatted for the post):

list0 = [ array([[ 1,  2,  3], [ 2,  3,  4]]), 
          array([[11, 12, 13], [12, 13, 14]]) ]

Replacing the assignment to a using a loop, weird things happen:

import numpy as np
a = np.empty((3), int)
list0 = []
for idx in range(4):    
    for i in range(3):
        a[i] = idx*10 + i
    print("idx =",idx,"; a =",a)
    list0.append(a)
print("list0 =",list0)

The second line tells Python the shape of the array used (in my original case it is a three-dimensional array). For verification the generated arrays named 'a' are printed out. Appending the newly filled arrays 'a' to 'list0' finally shows four times the last line.

idx = 0 ; a = [ 0  1  2]
idx = 1 ; a = [10 11 12]
idx = 2 ; a = [20 21 22]
idx = 3 ; a = [30 31 32]
list0 = [ array([30, 31, 32]), array([30, 31, 32]), 
          array([30, 31, 32]), array([30, 31, 32]) ] 

I suppose that 'list0' simply contains four times a pointer to the array 'a' which only exists in one instance / memory range.

So: How can I physically append (copy?) each of the different arrays 'a' to the list? Is it a python bug or is it simply my misunderstanding of something? Certainly I should think more pythonian ;c)

Thanks for help, Peter

Solution 1

The problem

You're appending the same array a to your list0 4 times. Arrays like a are mutable objects, which means, among other things, that when you assign values to them the underlying object changes. Since the array is present in your list 4 times, those changes (seem to) show up in 4 different places.

Solution

You can fix the code you have with one small change. Append a copy of the array to your list, instead of the array itself:

import numpy as np
a = np.empty((3), int)
list0 = []
for idx in range(4):    
    for i in range(3):
        a[i] = idx*10 + i
    print("idx =",idx,"; a =",a)
    list0.append(a.copy())
print("list0 =",list0)

Output:

idx = 0 ; a = [0 1 2]
idx = 1 ; a = [10 11 12]
idx = 2 ; a = [20 21 22]
idx = 3 ; a = [30 31 32]
list0 = [array([0, 1, 2]), array([10, 11, 12]), array([20, 21, 22]), array([30, 31, 32])]

Optimized solution

Python/Numpy offer many better ways (both in terms of using fewer lines of code and running faster) to initialize arrays. For a bunch of ranges like this, here is a reasonable approach:

list0 = [np.arange(n*10, n*10+3) for n in range(4)]
print(list0)

Output:

[array([0, 1, 2]), array([10, 11, 12]), array([20, 21, 22]), array([30, 31, 32])]

You might also consider just using a single 2D array in place of a list of arrays. One single array is often easier to work with than a heterogenous mix of arrays in a list. Here's how you do that:

arr0 = np.array([np.arange(n*10, n*10+3) for n in range(4)])
print(arr0)

Output:

[[ 0  1  2]
 [10 11 12]
 [20 21 22]
 [30 31 32]]

Solution 2

Just do this:

list_to_append.append(np_array.copy())

In a nutshell, numpy arrays or lists are mutable objects, which means that you when you assign a numpy array or list to a variable, what you are really assigning are references to memory locations aka pointers.

In your case, "a" is a pointer, so what you are really doing is appending to list0 an address to the memory location pointed by "a", and not the actual values pointed by the pointer. Thus it means that each new position of "list0", after appending, turns out to be the same address of memory: "a".

So, instead of:

list0.append(a)

You call the copy() method of "a" that creates a new memory location for the new values of "a" and returns it:

list0.append(a.copy())

Comments

[peets]

Many thanks tel (and others). Helped much in understanding and putting pointers out of my pymind. The .copy() worked fine at once. My data is not generated as in the example but received in a binary array from pyserial and is distributed to a 3D matrix. These are are stacked together for later evaluation. Looking forward to adapt it to a more numpy like and faster method.

[dspencer]

It would be good if you can explain why the OP's approach didn't behave as they expect, as this will also help future readers with this problem. You can then explain why your approach doesn't encounter such issues.

[Brian]

While this code may solve the question, including an explanation of how and why this solves the problem would really help to improve the quality of your post, and probably result in more up-votes. Remember that you are answering the question for readers in the future, not just the person asking now. Please edit your answer to add explanations and give an indication of what limitations and assumptions apply.

[peets]

Regarding requested explanations on obove answer: your suggestions are helpful, also for me, not to forget some explanation if I can answer a question. Good explanations were posted above; valuable thoughts for my deeper understanding and further reading. But Fsn9 s answer would have helped me in a second, arguing that pointers are not the way to think in Python ;c)

[Admin]

Thanks for the advice. I guess that it's better now.