How to change all the dictionary keys in a for loop with d.items()?

Question

I would like some help with understanding why this code is not working as expected.

If one wants to change the key of a dictionary but keep the value, he/she might use:

d[new_key] = d.pop[old_key]

I want to modify all the keys (and keep the values in place) but the code below skips certain lines - ("col2") remains untouched. Is it because dictionaries are unordered and I keep changing the values in it?

How would I go about changing the keys and keep the values without creating a new dictionary?

import time
import pprint

name_dict = {"col1": 973, "col2": "1452 29th Street",
             "col3": "Here is a value", "col4" : "Here is another value",
             "col5" : "NULL", "col6": "Scottsdale",
             "col7": "N/A", "col8" : "41.5946922",
             "col9": "Building", "col10" : "Commercial"}


for k, v in name_dict.items():
    print("This is the key: '%s' and this is the value '%s'\n" % (k, v) )
    new_key = input("Please enter a new key: ")
    name_dict[new_key] = name_dict.pop(k)
    time.sleep(4)

pprint.pprint(name_dict)

Solution 1

It's never a good idea to change the object you're iterating over. Normally dict even throws an exception when you attempt it:

name_dict = {1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6}

for k, v in name_dict.items():
    name_dict.pop(k)

RuntimeError: dictionary changed size during iteration

However in your case you add one item for every removed item. That makes it more convolved. To understand what's happening you need to know that a dictionary is somewhat like a sparse table. For example a dictionary like {1: 1, 3: 3, 5: 5} could look like this (this changed in Python 3.6, for 3.6 and newer the following isn't correct anymore):

hash    key    value
   -      -        - 
   1      1        1
   -      -        - 
   3      3        3
   -      -        - 
   5      5        5
   -      -        - 
   -      -        - 
   -      -        - 

That's also the order in which it is iterated. So in the first iteration it will go to the second item (where the 1: 1 is stored). Let's assume you change the key to 2 and remove the key 1 the dict would look like this:

hash    key    value
   -      -        - 
   -      -        - 
   2      2        1
   3      3        3
   -      -        - 
   5      5        5
   -      -        - 
   -      -        - 
   -      -        - 

But we're still at the second line, so the next iteration it will go to the next "not-empty" entry which is 2: 1. Oups ...

It's even more complicated with strings as keys because string hashes are randomized (on a per session basis) so the order inside the dictionary is unpredictable.

In 3.6 the internal layout was changed a bit but something similar happens here.

Assuming you have this loop:

name_dict = {1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6}

for k, v in name_dict.items():
    # print(k, k+6, name_dict.__sizeof__())
    name_dict[k+6] = name_dict.pop(k)
    # print(name_dict)

The initial layout is like this:

key   value
  1       1
  2       2
  3       3
  4       4
  5       5
  6       1

The first loop removes 1 but adds 7. Because dictionaries are ordered in 3.6 this inserts a placeholder where 1 had been:

key   value
  -       -
  2       2
  3       3
  4       4
  5       5
  6       1
  7       2

This goes on until you replace 4 with 10.

key   value
  -       -
  -       -
  -       -
  -       -
  5       5
  6       1
  7       2
  8       3
  9       4
 10       5

But when you replace 5 with 11 the dictionary will need to increase it's size. Then something special happens: The placeholders are removed:

key   value
  6       6
  7       1
  8       2
  9       3
 10       4
 11       5

So, we were at position 5 in the last iteration and now we change line 6. But line 6 contains 11: 5 right now. Oups...

Never change the object you're iterating over: Don't mess with the keys during iteration (values are okay)!

You could instead keep a "translation table" (don't know if that violates your "without creating a new dict" requirement but you need some kind of storage to make your code work correctly) and do the renaming after the loop:

translate = {}
for k, v in name_dict.items():
    print("This is the key: '%s' and this is the value '%s'\n" % (k, v) )
    new_key = input("Please enter a new key: ")
    translate[k] = new_key
    time.sleep(4)

for old, new in translate.items():
    name_dict[new] = name_dict.pop(old)

Solution 2

in python3 dict.items() is just a view on the dict. as you are not allowed to modify an iterable while iterating you are not allowed to modify a dict while iterating over dict.items(). you have to copy items() to a list before iterating

for k, v in list(name_dict.items()):
    ...
    name_dict[new_key] = name_dict.pop(k)

this does fulfill your 'no new dict' requirement, although the list holds in fact a full copy of all your data.

you could relax the memory footprint a little by copying just the keys

for k in list(name_dict):
    v = name_dict.pop(k)
    ...
    name_dict[new_key] = v

---

EDIT: credits to Sven Krüger, he raised the possibility of a old-key-new-key collision problem. in that case you have to go for

kv = list(name_dict.items())
name_dict.clear()
for k, v in kv :
    ...
    name_dict[new_key] = v

by the way, there is a use case for not creating a new dict, the current one might be referenced somwhere else.

Solution 3

In order to have a iterable object in your working memory which is not depending on your original dictionary you can use the method fromkeys. It is possible to now assign new keys with your old values. But there is one thing you have to keep in mind: You cannot assign a value to a new key that is not the certain old key while the new key is also another key from the old set of keys.

Old_Keys = { old_key_1, old_key_2, ..., old_key_n }

So you assign the value related to the old key to the new key.

old_key_1  ->  new_key_1 not in Old_Keys  # Okay!
old_key_2  ->  new_key_2 == old_key_4     # Boom!... Error!...

Be aware of this when you use the following!

CODE

D = {'key1': 'val1', 'key2': 'val2', 'key3': 'val3'}

for key in D.fromkeys(D) :
    new_key = raw_input("Old Key: %s, New Key: " % key)
    D[new_key] = D.pop(key)

print D

CONSOLE

Old Key: key1, New Key: abc

Old Key: key2, New Key: def

Old Key: key3, New Key: ghi

{"abc": 'val1', "def": 'val2', "ghi": 'val3'}

Comments

[RomanPerekhrest]

dictionaries are unordered - exactly

[Jon Clements]

Could just do... new_dict = {input('Enter new key for {}'.format(k)): v for k, v in name_dict.items()}... ?

[RomanPerekhrest]

@JonClements, without creating a new dictionary

[Chris_Rands]

Modifying while you iterate tends to lead to unexpected behavior, why don't you want a new dict, is memory the issue?

[Jon Clements]

@RomanPerekhrest ahh good point... but still... unless it's purely theoritical or an absolute requirement - it's no doubt the easiest...

[Jon Clements]

Okay so... name_dict.update((input('new key for {}'.format(k)), name_dict.pop(k)) for k in name_dict)... Might want to make that last bit list(name_dict) though... need more coffee first...

[Robert]

thank you all. It is not necessarily a memory issue. I just thought this issue is interesting to see if indeed should be solved in a different way and to make sure I understand the ups and downs of my learning process. I appreciate every reply and concern/opinion raised!

[MSeifert]

Why shouldn't you be "allowed" to modify an iterable while iterating? It often has (unwanted) side-effects when you change the iterable you're iterating over but that doesn't mean it's not allowed...

[stefan]

I agree, it is allowed fro language point of view. To me, it is not allowed from a design point of view unless you fully understand the side effects and you are the only maintainer ever :-). I would guess in python3 programmers error statistics this is one of the top scorers.

[Robert]

This is a very interesting learning experience. Much appreciate the time involved in helping a fellow Pyhtonista out. The way you presented the situations is perfect and easy to understand.

[stefan]

fromkeys actually creates a new dict, which is not allowed according to the question. the key collision is a good point.

[Sven Krüger]

My understanding is that the resulting dictionary is not allowed to be a new variable.