Spark DataFrame: How to specify schema when writing as Avro

Question

I want to write a DataFrame in Avro format using a provided Avro schema rather than Spark's auto-generated schema. How can I tell Spark to use my custom schema on write?

Solution

After applying the patch in https://github.com/databricks/spark-avro/pull/222/, I was able to specify a schema on write as follows:

df.write.option("forceSchema", myCustomSchemaString).avro("/path/to/outputDir")

Comments

[erwaman]

The "avroSchema" option has no effect for .write. It's only used for .read by the DefaultSource. See github.com/databricks/spark-avro/blob/….