How to make a completely unshared copy of a complicated list? (Deep copy is not enough)

Question

Have a look at this Python code:

a = [1, 2, 3]
b = [4, 5, 6]
c = [[a, b], [b, a]] # [[[1, 2, 3], [4, 5, 6]], [[4, 5, 6], [1, 2, 3]]]
c[0][0].append(99)   # [[[1, 2, 3, 99], [4, 5, 6]], [[4, 5, 6], [1, 2, 3, 99]]]

Notice how modifying one element of c modifies that everywhere. That is, if 99 is appended to c[0][0], it is also appended to c[1][1]. I am guessing this is because Python is cleverly referring to the same object for c[0][0] and c[1][1]. (That is their id() is the same.)

Question: Is there something that can be done to c so that its list elements can be safely locally modified? Above is just an example, my real problem has a list much more complicated, but having a similar problem.

(Sorry for the poorly formed question above. Python gurus please feel free to modify the question or tags to better express this query.)

Solution 1

To convert an existing list of lists to one where nothing is shared, you could recursively copy the list.

deepcopy will not be sufficient, as it will copy the structure as-is, keeping internal references as references, not copies.

def unshared_copy(inList):
    if isinstance(inList, list):
        return list( map(unshared_copy, inList) )
    return inList

alist = unshared_copy(your_function_returning_lists())

Note that this assumes the data is returned as a list of lists (arbitrarily nested). If the containers are of different types (eg. numpy arrays, dicts, or user classes), you may need to alter this.

Solution 2

When you want a copy, you explicitly make a copy - the cryptical [:] "slice it all" form is idiomatic, but my favorite is the much-more-readable approach of explicitly calling list.

If c is constructed in the wrong way (with references instead of shallow copies to lists you want to be able to modify independently) the best thing would be to fix the way it's built (why build it wrong and then labor to fix it?!), but if that's outside your control it IS possible to undo the damage -- just loop on c (recursively if needed), with an index, reassigning the relevant sublists to their copies. For example, if you know for sure that c's structure is two-level as you indicate, you can save yourself without recursion:

def fixthewronglymadelist(c):
  for topsublist in c:
    for i, L in enumerate(topsublist):
      topsublist[i] = list(L)

Despite what other answers suggest, copy.deepcopy would be hard to bend to this peculiar purpose, if all you're given is the wrongly made c: doing just copy.deepcopy(c) would carefully replicate whatever c's topology is, including multiple references to the same sublists! :-)

Solution 3

Depending on you situation, you might want to work against a deep copy of this list.

Solution 4

Use [:]:

>>> a = [1, 2]
>>> b = a[:]
>>> b.append(9)
>>> a
[1, 2]

Alteratively, use copy or deepcopy:

>>> import copy
>>> a = [1, 2]
>>> b = copy.copy(a)
>>> b.append(9)
>>> a
[1, 2]

copy works on objects other than lists. For lists, it has the same effect as a[:]. deepcopy attempts to recursively copy nested elements, and is thus a more "thorough" operation that copy.

Solution 5

To see Stephan's suggestion at work, compare the two outputs below:

a = [1, 2, 3]
b = [4, 5, 6]
c = [[a, b], [b, a]]
c[0][0].append(99)
print c
print "-------------------"
a = [1, 2, 3]
b = [4, 5, 6]
c = [[a[:], b[:]], [b[:], a[:]]]
c[0][0].append(99)
print c

The output is as follows:

[[[1, 2, 3, 99], [4, 5, 6]], [[4, 5, 6], [1, 2, 3, 99]]]
-------------------
[[[1, 2, 3, 99], [4, 5, 6]], [[4, 5, 6], [1, 2, 3]]]

View more solutions

Comments

[Kenan Banks]

When you get a half hour, you may want to study Python's Data Model: docs.python.org/reference/datamodel.html

[Markus Joschko]

Michael: Please remember that in this case I am getting c from a function over which I have no control. If you are just handed c, which might have many such lists inside it which refer to the same object, how would you deal with it?

[Markus Joschko]

Thanks for the suggestion. But, both (shallow) copy and deep copy did not work for my actual problem. I had to resort to Brian's solution.

[Markus Joschko]

Thanks Alex. Indeed, my actual list is more complicated (in construction) and deep copy did not solve the problem. I used a recursive unshared copy (Brian's answer) to solve the problem (since the list was 3 level).

[Markus Joschko]

Thanks Stephan202. Your solution works only for very simple lists and their construction. My actual list is way more complicated in construction and what solved my problem in the end was Brian's answer of a recursive unshared copy of the list.

[Markus Joschko]

Thanks Brian! Your intuition about the deep copy not working was right! Your answer solved my actual problem. I have also modified your code for Python 3, where map() returns a map object, not a list.

[TMWP]

In python 2.7, it is necessary to start with import copy before the commands will work, something the documentation does not tell you and which a newbie might not realize (thinking maybe this is builtin). In testing, this answer solved a rather complex similar problem I had where the list looked even more complex than the example given in this question. This suggests that using deep copy should be tried and used where it works and the accepted answer should be used when it won't. Simpler and more readable is always better, but only when it works.