Making a pandas dataframe from a .npy file

Question

I'm trying to make a pandas dataframe from a .npy file which, when read in using np.load, returns a numpy array containing a dictionary. My initial instinct was to extract the dictionary and then create a dataframe using pd.from_dict, but this fails every time because I can't seem to get the dictionary out of the array returned from np.load. It looks like it's just np.array([dictionary, dtype=object]), but I can't get the dictionary by indexing the array or anything like that. I've also tried using np.load('filename').item() but the result still isn't recognized by pandas as a dictionary.

Alternatively, I tried pd.read_pickle and that didn't work either.

How can I get this .npy dictionary into my dataframe? Here's the code that keeps failing...

import pandas as pd
import numpy as np
import os

targetdir = '../test_dir/'

filenames = []
successful = []
unsuccessful = []
for dirs, subdirs, files in os.walk(targetdir):
    for name in files:
        filenames.append(name)
        path_to_use = os.path.join(dirs, name)
        if path_to_use.endswith('.npy'):
            try:
                file_dict = np.load(path_to_use).item()
                df = pd.from_dict(file_dict)
                #df = pd.read_pickle(path_to_use)
                successful.append(path_to_use)
            except:
                unsuccessful.append(path_to_use)
                continue

print str(len(successful)) + " files were loaded successfully!"
print "The following files were not loaded:"
for item in unsuccessful:
    print item + "\n"

print df

Solution

Let's assume once you load the .npy, the item (np.load(path_to_use).item()) looks similar to this;

{'user_c': 'id_003', 'user_a': 'id_001', 'user_b': 'id_002'}

So, if you need to come up with a DataFrame like below using above dictionary;

  user_name user_id
0    user_c  id_003
1    user_a  id_001
2    user_b  id_002

You can use;

df = pd.DataFrame(list(x.item().iteritems()), columns=['user_name','user_id'])

---

If you have a list of dictionaries like below;

users = [{'u_name': 'user_a', 'u_id': 'id_001'}, {'u_name': 'user_b', 'u_id': 'id_002'}]

You can simply use

df = pd.DataFrame(users)

To come up with a DataFrame similar to;

     u_id  u_name
0  id_001  user_a
1  id_002  user_b

---

Seems like you have a dictionary similar to this;

data = {
    'Center': [[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]],
    'Vpeak': [1.1, 2.2],
    'ID': ['id_001', 'id_002']
}

In this case, you can simply use;

df = pd.DataFrame(data)  # df = pd.DataFrame(file_dict.item()) in your case

To come up with a DataFrame similar to;

    Center          ID      Vpeak
0   [0.1, 0.2, 0.3] id_001  1.1
1   [0.4, 0.5, 0.6] id_002  2.2

---

If you have ndarray within the dict, do some preprocessing similar to below; and use it to create the df;

for key in data:
    if isinstance(data[key], np.ndarray):
        data[key] = data[key].tolist()

df = pd.DataFrame(data)