pySpark Create DataFrame from RDD with Key/Value

Question

If I have an RDD of Key/Value (key being the column index) is it possible to load it into a dataframe? For example:

(0,1), (0,1), (0,2), (1,2), (1,10), (1,20), (3,18), (3,18), (3,18)

And have the dataframe look like:

1,2,18
1,10,18
2,20,18

Solution

Yes it's possible (tested with Spark 1.3.1) :

>>> rdd = sc.parallelize([(0,1), (0,1), (0,2), (1,2), (1,10), (1,20), (3,18), (3,18), (3,18)])
>>> sqlContext.createDataFrame(rdd, ["id", "score"])
Out[2]: DataFrame[id: bigint, score: bigint]

Comments

[Frozen Flame]

Is this equivolent to rdd.toDF( ["id", "score"])?

[Jack Daniel]

'RDD' object has no attribute 'toDF' . Facing this error

[Jack Daniel]

I am using 1.6 spark and pyspark. Unable to load the sql.SQLContext and create DataFrame out of it.