Pandas merge giving error "Buffer has wrong number of dimensions (expected 1, got 2)"

Question

I am trying to do a pandas merge and get the above error from the title when I try to run it. I am using 3 columns to match on whereas just before I do similar merge on only 2 columns and it works fine.

df = pd.merge(df, c, how="left",
        left_on=["section_term_ps_id", "section_school_id", "state"],
        right_on=["term_ps_id", "term_school_id", "state"])

columns for the two dataframes

df:

Index([u'section_ps_id', u'section_school_id', u'section_course_number', u'secti
on_term_ps_id', u'section_staff_ps_id', u'section_number', u'section_expression'
, u'section_grade_level', u'state', u'sections_id', u'course_ps_id', u'course_sc
hool_id', u'course_number', u'course_schd_dept', u'courses_id', u'school_ps_id',
 u'course_school_id', u'school_name', u'school_abbr', u'school_low_grade', u'sch
ool_high_grade', u'school_alt_school_number', u'school_state', u'school_phone',
u'school_fax', u'school_principal', u'school_principal_phone', u'school_principa
l_email', u'school_asst_principal', u'school_asst_principal_phone', u'school_ass
t_principal_email'], dtype='object')

c:

Index([u'term_ps_id', u'term_school_id', u'term_portion',
u'term_start_date', u' term_end_date', u'term_abbreviation',
u'term_name', u'state', u'terms_id', u'sch ool_ps_id',
u'term_school_id', u'school_name', u'school_abbr', u'school_low_grad
e', u'school_high_grade', u'school_alt_school_number',
u'school_state', u'school
_phone', u'school_fax', u'school_principal', u'school_principal_phone', u'school
_principal_email', u'school_asst_principal', u'school_asst_principal_phone', u's chool_asst_principal_email'],
dtype='object')

Is it possible to merge on three columns like this? Is there anything wrong from the merge call here?

Solution 1

As mentioned in the comments, you have a dupe column:

Solution 2

This Will remove the duplicated columns from the Dataframe

df = df[list(df.columns[~df.columns.duplicated()])]

Solution 3

To adress the issue of the dupe columns you can either drop the dupe column using duplicated with smth. like:

c = c[~c.columns.duplicated(keep='first')]

or adding an additional char to either one of the DataFrames using for example: c.columns=[c.columns[i]+str(i) for i in range(len(c.columns))]

Keep in mind that in this case you must adjust the merging part

Comments

[user1017373]

How to remove the duplicated column..?

[Rocketq]

@user1017373 drop it by index)

[braunmagrin]

@Rocketq your suggestion should be included in the answer, as it is the actual way of fixing the problem ;)

[Timus]

Welcome to SO! Are you aware that this question is almonst 6 years old (and has several answers, including an accepted one)? And are you sure the duplicate problem is solved by simply upgrading?

[Aman Saini]

@Timus Thanks :)... I have mentioned that this may work IF there are no duplicate columns.

[Timus]

But the dupes are the problem ... ?