how to get the column names and their datatypes of parquet file using pyspark?

14,245

Solution 1

You can simply read the file and use schema to access individual fields:

sqlContext.read.parquet(path_to_parquet_file).schema.fields

Solution 2

Use dataframe.printSchema() - Prints out the schema in the tree format.

df.printSchema() root |-- age: integer (nullable = true) |-- name: string (nullable = true)

You can redirect the output of your program and capture that in a text file.

Share:
14,245
Shubham Mishra
Author by

Shubham Mishra

Updated on June 20, 2022

Comments

  • Shubham Mishra
    Shubham Mishra over 1 year

    i have a parquet file on my hadoop cluster ,i want to capture the column names and their datatypes and write it on a textfile.how to get the column names and their datatypes of parquet file using pyspark.