Pyspark: spark data frame column width configuration in Jupyter Notebook
15,299
Solution 1
I don't think you can set a specific width, but this will ensure your data is not cutoff no matter the size
my_df.select('field_1','field_2').show(10, truncate = False)
Solution 2
This should give you what you want
import pandas as pd
pd.set_option('display.max_colwidth', 80)
my_df.select('field_1','field_2').limit(100).toPandas()
Author by
Edamame
Updated on June 30, 2022Comments
-
Edamame almost 2 years
I have the following code in Jupyter Notebook:
import pandas as pd pd.set_option('display.max_colwidth', 80) my_df.select('field_1','field_2').show()
I want to increase the column width so I could see the full value of
field_1
andfield_2
. I know we can usepd.set_option('display.max_colwidth', 80)
for pandas data frame, but it doesn't seem to work for spark data frame.Is there a way to increase the column width for the spark data frame like what we did for pandas data frame? Thanks!