Python - Filter list of dictionaries based on multiple keys

Question

Say I have the dictionary:

myDict = [{'first': 'James', 'middle': 'Smith', 'last': 'Joule'}, 
        {'first': 'James', 'middle': 'Johnson', 'last': 'Watt'},
        {'first': 'Christian', 'middle': 'Edward', 'last': 'Doppler'}
        {'first': 'Robert', 'last': 'Antonio'}]

And I have a list called keys:

keys = ["middle", "last"]

I want to filter myDict based on each value in keys, which would result in

filteredDict = [{'middle': 'Smith', 'last': 'Joule'},
              {'middle': 'Johnson', 'last': 'Watt'},
              {'middle': 'Edward', 'last': 'Doppler'},
              {'last': 'Antonio'}]

As seen in the list dictionary in myDict list, the dictionary DOESNT have to have both keys in order to be placed into filteredDict. Is there an easy way to do this with dictionary comprehension in Python?

Solution 1

With list comprehensions:

myDict = [{'first': 'James', 'middle': 'Smith', 'last': 'Joule'}, 
      {'first': 'James', 'middle': 'Johnson', 'last': 'Watt'},
      {'first': 'Christian', 'middle': 'Edward', 'last': 'Doppler'},
      {'first': 'Robert', 'last': 'Antonio'}]

keys = {"middle", "last"}

l = [{k:v for k, v in i.items() if k in keys} for i in myDict]

But you can also use map for this:

myDict = [{'first': 'James', 'middle': 'Smith', 'last': 'Joule'}, 
      {'first': 'James', 'middle': 'Johnson', 'last': 'Watt'},
      {'first': 'Christian', 'middle': 'Edward', 'last': 'Doppler'},
      {'first': 'Robert', 'last': 'Antonio'}]

keys = {"middle", "last"}

l = list(map(lambda x: {k:v for k, v in x.items() if k in keys}, myDict))
print(l)

output:

[{'last': 'Joule', 'middle': 'Smith'}, {'last': 'Watt', 'middle': 'Johnson'}, {'last': 'Doppler', 'middle': 'Edward'}, {'last': 'Antonio'}]

Solution 2

Use neverwalkaloner's answer if you are just doing this once. But, if you find yourself manipulating lists of dictionaries often, I've written a free library called PLOD that streamlines much of this.

>>> from PLOD import PLOD
>>> l = PLOD(myDict).dropKey("middle").returnList()
>>> l
[{'last': 'Joule', 'first': 'James'}, {'last': 'Watt', 'first': 'James'}, {'last': 'Doppler', 'first': 'Christian'}, {'last': 'Antonio', 'first': 'Robert'}]
>>> print(PLOD(l).returnString())
[
    {first: 'James'    , last: 'Joule'  },
    {first: 'James'    , last: 'Watt'   },
    {first: 'Christian', last: 'Doppler'},
    {first: 'Robert'   , last: 'Antonio'}
]
>>> 

The library is on PyPi: https://pypi.python.org/pypi/PLOD

To more universally do what you want, I'd need to add a new class method. Perhaps .filterKeys. Perhaps I'll do that in version 1.8. It would then go:

>>> l = PLOD(myDict).filterKeys(['first', 'last']).returnList()

Hmmm...

BTW, the lib supports Python 2.7.x right now. Still working on the 3.5.x release.

Comments

[volcano]

I would just turn keys into set - to speed up lookup

[K. A. Buhr]

Also, it looks a little cleaner to use a list comprehension in place of map, so just write [{ k: v for (k,v) in x.items() if k in keys } for x in myDict]

[volcano]

BTW, map is another form of comprehension. And since in Python 3 it produces map object - which is essentially iterator, comprehension will be universal choice.

[akash karothiya]

working perfect, just wanted to check which one takes less time to execute

[neverwalkaloner]

@K.A.Buhr you are right but for some reason I thought about it after map.

[ScipioAfricanus]

Shouldnt keys be a list @op?