Good alternative to Pandas .append() method, now that it is being deprecated?

Question

I use the following method a lot to append a single row to a dataframe. One thing I really like about it is that it allows you to append a simple dict object. For example:

# Creating an empty dataframe
df = pd.DataFrame(columns=['a', 'b'])

# Appending a row
df = df.append({ 'a': 1, 'b': 2 }, ignore_index=True)

Again, what I like most about this is that the code is very clean and requires very few lines. Now I suppose the recommended alternative is:

# Create the new row as its own dataframe
df_new_row = pd.DataFrame({ 'a': [1], 'b': [2] })
df = pd.concat([df, df_new_row])

So what was one line of code before is now two lines with a throwaway variable and extra cruft where I create the new dataframe. :( Is there a good way to do this that just uses a dict like I have in the past (that is not deprecated)?

Solution 1

Create a list with your dictionaries, if they are needed, and then create a new dataframe with df = pd.DataFrame.from_records(your_list). List's "append" method are very efficient and won't be ever deprecated. Dataframes on the other hand, frequently have to be recreated and all data copied over on appends, due to their design - that is why they deprecated the method

Solution 2

I also like the append method. But you can do it in one line with a list of dicts

df = pd.concat([df, pd.DataFrame.from_records([{ 'a': 1, 'b': 2 }])])

or using loc and tuples for values on DataFrames with incremenal ascending indexes

df.loc[len(df), ['a','b']] = 1, 2

or maybe

df.loc[len(df), df.columns] = 3, 4

Comments

[Paul Rougieux]

pandas issue 35407 explains that df.append was deprecated because: "Series.append and DataFrame.append [are] making an analogy to list.append, but it's a poor analogy since the behavior isn't (and can't be) in place. The data for the index and values needs to be copied to create the result."

[Ben Watson]

Came across this warning today. However when I used concat as the alternative I got "cannot concatenate object of type '<class 'list'>'; only Series and DataFrame objs are valid". So frustrating.....

[zby]

How do you know that it is deprecated? At pandas.pydata.org/docs/reference/api/… (which currently shows version 1.4.0) I don't see any mention about that. Even at the dev tree I don't see any deprecation warning: pandas.pydata.org/docs/dev/reference/api/…

[tgrandje]

I agree ; though when you use append method (with 1.4.0) you run into a "FutureWarning : The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead". You will find the details in the "what's new" page

[Paul Rougieux]

@zby the update to the documentation is being dealt with in this pull request: github.com/pandas-dev/pandas/pull/45587

[Rafael Gaitan]

You can also use ignore_index df = pd.concat([df, pd.DataFrame.from_records([{ 'a': 1, 'b': 2 }])], ignore_index=True)

[kush]

this brought a ten times faster speed to my code , thanks so much man

[jsbueno]

That is actually the reason they are deprecating df.append. Thank the Pandas maintainers for that. Still, the "new way to do it" should be more proeminent in their docs, for sure.

[Alex Punnen]

creating a dataframe from a huge list took too much time compared to getting data and appending in chunks to the dataframe. However the pd.concat method below worked just fine