How to replace NaN from Dictionary in python

Question

I have the following code:

my_dataset={'item1':{'item11':0,'item12':'NaN','item13':2},'item2':{'item21':0,'item22':'NaN','item23':2}}
my_dataset_clean=my_dataset
for item in my_dataset:
     my_dataset_clean[item] = {k: 0 for k in my_dataset[item] if isnan(my_dataset[item][k])}

I am getting this error:

my_dataset_clean[item] = {k: 0 for k in my_dataset[item] if isnan(my_dataset[item][k])} TypeError: must be real number, not str

Any ideas on how to solve the issue? I want to replace NaN with 0s

Solution 1

You need to check 'NaN' as a string instead of use isnan (presumably np.isnan) which expects a numeric value. But there are a few other points worth noting:

1. You don't define what happens when your value is not 'NaN'. Without this you will lose the other items. See below for how you can define a ternary statement.
2. my_dataset_clean = my_dataset does not create a copy of your dictionary. You need to be explicit, e.g. my_dataset_clean = my_dataset.copy(), to do this. Otherwise you just have 2 variables pointing to the same dictionary.
3. You can use a dictionary comprehension to avoid having to make a copy of your dictionary.

Here's an example with a dictionary comprehension and ternary statement:

my_dataset_clean = {k: {k2: 0 if v2 == 'NaN' else v2 for k2, v2 in v.items()} \
                    for k, v in my_dataset.items()}

Here k, v refer to keys / values in the outer dictionary, and k2, v2 refer to keys / values in inner dictionaries.

Solution 2

'NaN' in your dictionaries is a string. Instead of isnan you can just use string comparison: my_dataset[item][k] == 'NaN'.

Comments

[fuagda]

Many thanks for the 3 points. Your example is exactly what I wanted to achieved