Save Spark RDD to Hive Table

Question

In spark I want to save RDD objects to hive table. I am trying to use createDataFrame but that is throwing

Exception in thread "main" java.lang.NullPointerException

 val products=sc.parallelize(evaluatedProducts.toList);
 //here products are RDD[Product]
 val productdf = hiveContext.createDataFrame(products, classOf[Product])

I am using Spark 1.5 version.

Solution

If your Product is a class (not a case class), I suggest you transform your rdd to RDD[Tuple] before creating the DataFrame:

import org.apache.spark.sql.hive.HiveContext

val hiveContext = new HiveContext(sc)
import hiveContext.implicits._

val productDF = products
  .map({p: Product => (p.getVal1, p.getVal2, ...)})
  .toDF("col1", "col2", ...)

With this approach, you will have the Product attributes as columns in the DataFrame.

Then you can create a temp table with:

productDF.registerTempTable("table_name")

or a physical table with:

productDF.write.saveAsTable("table_name")