pandas.concat: Cannot handle a non-unique multi-index! Pandas Python

Question

I am trying to concatenate 100 dataframes that have 2 datetime indexes using the following code:

concat_df = pd.concat([df_dict[c] for c in df_dict], axis = 1)

But somewhere one of the dataframes (I assume it is one but it could be more) is causing the following exception to occur:

Exception: cannot handle a non-unique multi-index!

Any ideas why?

Is it referring to the first index or the second index?

Solution 1

probably the following option ignore_index=True of concat helps.

not sure if this could be use like follows:

concat_df = pd.concat([df_dict[c] for c in df_dict], axis = 1,ignore_index=True)

to try!

Solution 2

I found it was referring to the first index my solution was: (thought not sure how efficient it is but the concat works afterwards)

dup_first_index_dates = np.where(np.array([np.sum(df_dict[c].index.duplicated()) for c in df_dict]) == 1)[0]
key = df_dict.keys()
for i in dup_first_index_dates :
    df_dict[key[i]] = df_dict[tickers[i]].reset_index().drop_duplicates('Level1').set_index(['Level1', 'Level2'])