Pandas: Setting no. of max rows

Question

I have a problem viewing the following DataFrame:

n = 100
foo = DataFrame(index=range(n))
foo['floats'] = np.random.randn(n)
foo

The problem is that it does not print all rows per default in ipython notebook, but I have to slice to view the resulting rows. Even the following option does not change the output:

pd.set_option('display.max_rows', 500)

Does anyone know how to display the whole array?

Solution 1

Set display.max_rows:

pd.set_option('display.max_rows', 500)

For older versions of pandas (<=0.11.0) you need to change both display.height and display.max_rows.

pd.set_option('display.height', 500)
pd.set_option('display.max_rows', 500)

See also pd.describe_option('display').

You can set an option only temporarily for this one time like this:

from IPython.display import display
with pd.option_context('display.max_rows', 100, 'display.max_columns', 10):
    display(df) #need display to show the dataframe when using with in jupyter
    #some pandas stuff

You can also reset an option back to its default value like this:

pd.reset_option('display.max_rows')

And reset all of them back:

pd.reset_option('all')

Solution 2

Personally, I like setting the options directly with an assignment statement as it is easy to find via tab completion thanks to iPython. I find it hard to remember what the exact option names are, so this method works for me.

For instance, all I have to remember is that it begins with pd.options

pd.options.<TAB>

Most of the options are available under display

pd.options.display.<TAB>

From here, I usually output what the current value is like this:

pd.options.display.max_rows
60

I then set it to what I want it to be:

pd.options.display.max_rows = 100

Also, you should be aware of the context manager for options, which temporarily sets the options inside of a block of code. Pass in the option name as a string followed by the value you want it to be. You may pass in any number of options in the same line:

with pd.option_context('display.max_rows', 100, 'display.max_columns', 10):
    some pandas stuff

You can also reset an option back to its default value like this:

pd.reset_option('display.max_rows')

And reset all of them back:

pd.reset_option('all')

It is still perfectly good to set options via pd.set_option. I just find using the attributes directly is easier and there is less need for get_option and set_option.

Solution 3

pd.set_option('display.max_rows', 500)
df

Does not work in Jupyter!
Instead use:

pd.set_option('display.max_rows', 500)
df.head(500)

Solution 4

It was already pointed in this comment and in this answer, but I'll try to give a more direct answer to the question:

from IPython.display import display
import numpy as np
import pandas as pd

n = 100
foo = pd.DataFrame(index=range(n))
foo['floats'] = np.random.randn(n)

with pd.option_context("display.max_rows", foo.shape[0]):
    display(foo)

pandas.option_context is available since pandas 0.13.1 (pandas 0.13.1 release notes). According to this,

[it] allow[s] you to execute a codeblock with a set of options that revert to prior settings when you exit the with block.

Solution 5

to set unlimited number of rows use

None

i.e.,

pd.set_option('display.max_columns', None)

now the notebook will display all the rows in all datasets within the notebook ;)

Similarly you can set to show all columns as

pd.set_option('display.max_rows', None)

now if you use run the cell with only dataframe with out any head or tail tags as

df

then it will show all the rows and columns in the dataframe df

View more solutions

Comments

[BubbleGuppies]

When I run your code in a default (i.e. no special configuration profile) notebook, I get a pretty printed table that is scrollable with all values. FYI, my pandas.__version__ = 0.9.1 (not sure if this matters)

[Ryan Saxe]

I meant regular shell, not ipython

[Andy Hayden]

I have a feeling this might be a bug in 0.11+...

[Andy]

Hi Andy. Has this already been confirmed by Wes? Where can I file this bug? Is there a workaround?

[Andy Hayden]

I just filed it here, I know there were some last minute changes in 0.11 to the DataFrame repr so I cc'd those in the bug report. Will let you know re workaround.

[Ted Petrou]

For those interested in setting options directly by their attributes, look at this answer below.

[nom-mon-ir]

+1 for the pd.describe_option('display'), I did not know all the options

[hanleyhansen]

Height is now deprecated so the display.max_rows option is enough.

[simtim]

You shouldn't be converting it to string. It's not what Andy asked for.

[Ninjakannon]

@simtim Andy asked how to "display the whole array". This will do that and is much simpler than the accepted answer.

[ijoseph]

with pd.option_context is the cleanest method among these answers; least side effects.

[BallpointBen]

For anyone only looking at the accepted answer: use with pd.option_context('display.height', 500, 'display.max_rows', 500): to only set these temporarily.

[MGB.py]

I should correct or give the best way to achieve this. Use None and don't limit to 500. #Temporary display all rows and columns with pd.option_context('display.max_rows',None, 'display.max_columns', None): display(df_facilities) The above code will take effect only in the cell containing the code so no need to reset in other cells.

[antonavy]

Not sure if it's a jupyter lab problem, but none of this is working. Only 'display.max_rows' = None does.

[Jan Joswig]

@antonavy I don't know if that is your issue but when setting display.max_rows to an integer, setting also display.min_rows is needed to affect how many rows are displayed in a truncated view (when max_rows is exceeded).

[Christoph Böddeker]

In github.com/pandas-dev/pandas/issues/… is an explanation, why that happens. You have to set also display.min_rows, because display.max_rows is the threshold, when you display display.min_rows number of roles. I have no idea, how they came up with these names.

[igorkf]

Isn't it max_columns ?

[arun]

Thanks @igorkf for update, I've made the changes Thanks for reading my post, happy learning

[questionto42standswithUkraine]

The first line now seems to work in jupyter notebook without the head() trick.

[Roland Pihlakas]

In newest versions of Pandas setting the display.height raises an exception now.

[wellplayed]

I guess it depends on what you are doing in your notebook, but I did find the context manager so useful that I wrapped it in a function and used display(df) within.

[Antonio Sesto]

This is terrible, unusable. Would it be so difficult to add a parameter display(this, n_rows=2 zillions)?