Pandas cannot read parquet files created in PySpark
Solution 1
Since this still seems to be an issue even with newer pandas versions, I wrote some functions to circumvent this as part of a larger pyspark helpers library:
import pandas as pd
import datetime
import os
def read_parquet_folder_as_pandas(path, verbosity=1):
files = [f for f in os.listdir(path) if f.endswith("parquet")]
if verbosity > 0:
print("{} parquet files found. Beginning reading...".format(len(files)), end="")
start = datetime.datetime.now()
df_list = [pd.read_parquet(os.path.join(path, f)) for f in files]
df = pd.concat(df_list, ignore_index=True)
if verbosity > 0:
end = datetime.datetime.now()
print(" Finished. Took {}".format(end-start))
return df
def read_parquet_as_pandas(path, verbosity=1):
"""Workaround for pandas not being able to read folder-style parquet files.
"""
if os.path.isdir(path):
if verbosity>1: print("Parquet file is actually folder.")
return read_parquet_folder_as_pandas(path, verbosity)
else:
return pd.read_parquet(path)
This assumes that the relevant files in the parquet "file", which is actually a folder, end with ".parquet". This works for parquet files exported by databricks and might work with others as well (untested, happy about feedback in the comments).
The function read_parquet_as_pandas()
can be used if it is not known beforehand whether it is a folder or not.
Solution 2
The problem is that Spark partitions the file due to its distributed nature (each executor writes a file inside the directory that receives the filename). This is not something supported by Pandas, which expects a file, not a path.
You can circumvent this issue in different ways:
-
Reading the file with an alternative utility, such as the
pyarrow.parquet.ParquetDataset
, and then convert that to Pandas (I did not test this code).arrow_dataset = pyarrow.parquet.ParquetDataset('path/myfile.parquet') arrow_table = arrow_dataset.read() pandas_df = arrow_table.to_pandas()
-
Another way is to read the separate fragments separately and then concatenate them, as this answer suggest: Read multiple parquet files in a folder and write to single csv file using python
Solution 3
If the parquet file has been created with spark, (so it's a directory) to import it to pandas use
from pyarrow.parquet import ParquetDataset
dataset = ParquetDataset("file.parquet")
table = dataset.read()
df = table.to_pandas()
Comments
-
Thomas over 2 years
I am writing a parquet file from a Spark DataFrame the following way:
df.write.parquet("path/myfile.parquet", mode = "overwrite", compression="gzip")
This creates a folder with multiple files in it.
When I try to read this into pandas, I get the following errors, depending on which parser I use:
import pandas as pd df = pd.read_parquet("path/myfile.parquet", engine="pyarrow")
PyArrow:
File "pyarrow\error.pxi", line 83, in pyarrow.lib.check_status
ArrowIOError: Invalid parquet file. Corrupt footer.
fastparquet:
File "C:\Program Files\Anaconda3\lib\site-packages\fastparquet\util.py", line 38, in default_open return open(f, mode)
PermissionError: [Errno 13] Permission denied: 'path/myfile.parquet'
I am using the following versions:
- Spark 2.4.0
- Pandas 0.23.4
- pyarrow 0.10.0
- fastparquet 0.2.1
I tried gzip as well as snappy compression. Both do not work. I of course made sure that I have the file in a location where Python has permissions to read/write.
It would already help if somebody was able to reproduce this error.
-
Thomas over 5 yearsThank you for your answer. It seems that reading single files (your second bullet point) works. However, the first thing does not work - it looks like pyarrow cannot handle PySpark's footer (see error message in question)
-
martinarroyo over 5 years@Thomas, I am unfortunately not sure about the footer issue.
-
Omkar Neogi almost 5 yearsOr you could try calling coalesce on the dataframe:
coalesce(1)
so it coalesces all the part files into one file and then read from the single file instead of a directory of files? -
Thomas over 4 years@OmkarNeogi: This is only possible if you are the person writing the files, not if you receive them from somebody else...
-
Tom N Tech over 3 yearsI updated this to work with the actual APIs, which is that you create a Dataset, convert it to a Table and then to a Pandas DataFrame.