Converting a Scala Iterable[tuple] to RDD

Question

I have a list of tuples, (String, String, Int, Double) that I want to convert to Spark RDD.

In general, how do I convert a Scala Iterable[(a1, a2, a3, ..., an)] into a Spark RDD?

Solution

There are a few ways to do this, but the most straightforward way is just to use Spark Context:

import org.apache.spark._
import org.apache.spark.rdd._
import org.apache.spark.SparkContext._

sc.parallelize(YourIterable.toList)

I think sc.Parallelize needs a conversion to List, but it will preserve your structure, thus you will still get a RDD[String,String,Int,Double]