Converting a Pandas GroupBy output from Series to DataFrame

Question

I'm starting with input data like this

df1 = pandas.DataFrame( { 
    "Name" : ["Alice", "Bob", "Mallory", "Mallory", "Bob" , "Mallory"] , 
    "City" : ["Seattle", "Seattle", "Portland", "Seattle", "Seattle", "Portland"] } )

Which when printed appears as this:

   City     Name
0   Seattle    Alice
1   Seattle      Bob
2  Portland  Mallory
3   Seattle  Mallory
4   Seattle      Bob
5  Portland  Mallory

Grouping is simple enough:

g1 = df1.groupby( [ "Name", "City"] ).count()

and printing yields a GroupBy object:

                  City  Name
Name    City
Alice   Seattle      1     1
Bob     Seattle      2     2
Mallory Portland     2     2
        Seattle      1     1

But what I want eventually is another DataFrame object that contains all the rows in the GroupBy object. In other words I want to get the following result:

                  City  Name
Name    City
Alice   Seattle      1     1
Bob     Seattle      2     2
Mallory Portland     2     2
Mallory Seattle      1     1

I can't quite see how to accomplish this in the pandas documentation. Any hints would be welcome.

Solution 1

g1 here is a DataFrame. It has a hierarchical index, though:

In [19]: type(g1)
Out[19]: pandas.core.frame.DataFrame

In [20]: g1.index
Out[20]: 
MultiIndex([('Alice', 'Seattle'), ('Bob', 'Seattle'), ('Mallory', 'Portland'),
       ('Mallory', 'Seattle')], dtype=object)

Perhaps you want something like this?

In [21]: g1.add_suffix('_Count').reset_index()
Out[21]: 
      Name      City  City_Count  Name_Count
0    Alice   Seattle           1           1
1      Bob   Seattle           2           2
2  Mallory  Portland           2           2
3  Mallory   Seattle           1           1

Or something like:

In [36]: DataFrame({'count' : df1.groupby( [ "Name", "City"] ).size()}).reset_index()
Out[36]: 
      Name      City  count
0    Alice   Seattle      1
1      Bob   Seattle      2
2  Mallory  Portland      2
3  Mallory   Seattle      1

Solution 2

I want to slightly change the answer given by Wes, because version 0.16.2 requires as_index=False. If you don't set it, you get an empty dataframe.

Source:

Aggregation functions will not return the groups that you are aggregating over if they are named columns, when as_index=True, the default. The grouped columns will be the indices of the returned object.

Passing as_index=False will return the groups that you are aggregating over, if they are named columns.

Aggregating functions are ones that reduce the dimension of the returned objects, for example: mean, sum, size, count, std, var, sem, describe, first, last, nth, min, max. This is what happens when you do for example DataFrame.sum() and get back a Series.

nth can act as a reducer or a filter, see here.

import pandas as pd

df1 = pd.DataFrame({"Name":["Alice", "Bob", "Mallory", "Mallory", "Bob" , "Mallory"],
                    "City":["Seattle","Seattle","Portland","Seattle","Seattle","Portland"]})
print df1
#
#       City     Name
#0   Seattle    Alice
#1   Seattle      Bob
#2  Portland  Mallory
#3   Seattle  Mallory
#4   Seattle      Bob
#5  Portland  Mallory
#
g1 = df1.groupby(["Name", "City"], as_index=False).count()
print g1
#
#                  City  Name
#Name    City
#Alice   Seattle      1     1
#Bob     Seattle      2     2
#Mallory Portland     2     2
#        Seattle      1     1
#

EDIT:

In version 0.17.1 and later you can use subset in count and reset_index with parameter name in size:

print df1.groupby(["Name", "City"], as_index=False ).count()
#IndexError: list index out of range

print df1.groupby(["Name", "City"]).count()
#Empty DataFrame
#Columns: []
#Index: [(Alice, Seattle), (Bob, Seattle), (Mallory, Portland), (Mallory, Seattle)]

print df1.groupby(["Name", "City"])[['Name','City']].count()
#                  Name  City
#Name    City                
#Alice   Seattle      1     1
#Bob     Seattle      2     2
#Mallory Portland     2     2
#        Seattle      1     1

print df1.groupby(["Name", "City"]).size().reset_index(name='count')
#      Name      City  count
#0    Alice   Seattle      1
#1      Bob   Seattle      2
#2  Mallory  Portland      2
#3  Mallory   Seattle      1

The difference between count and size is that size counts NaN values while count does not.

Solution 3

The key is to use the reset_index() method.

Use:

import pandas

df1 = pandas.DataFrame( { 
    "Name" : ["Alice", "Bob", "Mallory", "Mallory", "Bob" , "Mallory"] , 
    "City" : ["Seattle", "Seattle", "Portland", "Seattle", "Seattle", "Portland"] } )

g1 = df1.groupby( [ "Name", "City"] ).count().reset_index()

Now you have your new dataframe in g1:

Solution 4

Simply, this should do the task:

import pandas as pd

grouped_df = df1.groupby( [ "Name", "City"] )

pd.DataFrame(grouped_df.size().reset_index(name = "Group_Count"))

Here, grouped_df.size() pulls up the unique groupby count, and reset_index() method resets the name of the column you want it to be. Finally, the pandas Dataframe() function is called upon to create a DataFrame object.

Solution 5

Maybe I misunderstand the question but if you want to convert the groupby back to a dataframe you can use .to_frame(). I wanted to reset the index when I did this so I included that part as well.

example code unrelated to question

df = df['TIME'].groupby(df['Name']).min()
df = df.to_frame()
df = df.reset_index(level=['Name',"TIME"])

View more solutions

Comments

[Ben]

I think this is the easiest way - a one liner which uses the nice fact that you can name the series column with reset_index: df1.groupby( [ "Name", "City"]).size().reset_index(name="count")

[Nehal J Wani]

You could have used: df1.groupby( [ "Name", "City"] ).size().to_frame(name = 'count').reset_index()

[Roman Pekar]

Is there a reason why as_index=False' stopped working in latest versions? I also tried to run df1.groupby(["Name", "City"], as_index=False ).size()` but it doesn't affect result (probably because result of the grouping is Series not DataFrame

[jezrael]

I am not sure, but it seems there are only 2 columns and groupby by these columns. But I am not sure, because I am not pandas developer.

[Alexander]

The second example using .reset_index() seems to me to be the best way of joining the output you will get from df.groupby('some_column').apply(your_custom_func). This was not intuitive for me.

[Sealander]

Check out the .to_frame() method: grouped_df.size().to_frame('Group_Count')

[John Strood]

Why add_suffix though?

[Adrian Keister]

Is this also true in Python 3? I'm finding a groupby function returning the pandas.core.groupby.DataFrameGroupBy object, not pandas.core.frame.DataFrame.

[matanster]

This answer seems irrelevant for latest python and pandas

[JeffZheng]

Could you please share the dataset you used for your solution? Thanks a lot!

[Excel Help]

.to_frame() is what I came here for and was the method I wasn't aware of, and it perfectly answers the question as it is currently worded. In my case, I wanted to keep my MultiIndex but just turn my resulting GroupBy Series into a DataFrame so Jupyter would display it nicely.

[Golden Lion]

I used the resulting dataframe converted from the groupby in a swarmplot sns.swarmplot(x='my_category', y='my_values', data=new_df)

[Sajid]

This works, thanks! Just a clarification, count() function counts all distinct values, thus skips duplicates automatically. After that, reset_index() does the trick of creating a new dataframe free from duplicates.

[young_souvlaki]

Use as_index=False. See answer by @jezrael.

[kilgoretrout]

reset_index doesn't have a name argument.

[Heath Raftery]

I was struck by the name argument too. Turns out the key is that DataFrameGroupBy.size() and friends return a Series by default, not a DataFrame. The reset_index() method on a Series does have name. The default return type can be changed by the as_index argument to groupby(). This loose typing and indirect method calling makes the document very hard to browse!

[artemis]

@NehalJWani that worked for me after hours of searching, thank you.