PySpark: Randomize rows in dataframe

Question

I have a dataframe and I want to randomize rows in the dataframe. I tried sampling the data by giving a fraction of 1, which didn't work (interestingly this works in Pandas).

Solution

It works in Pandas because taking sample in local systems is typically solved by shuffling data. Spark from the other hand avoids shuffling by performing linear scans over the data. It means that sampling in Spark only randomizes members of the sample not an order.

You can order DataFrame by a column of random numbers:

from pyspark.sql.functions import rand 

df = sc.parallelize(range(20)).map(lambda x: (x, )).toDF(["x"])
df.orderBy(rand()).show(3)

## +---+
## |  x|
## +---+
## |  2|
## |  7|
## | 14|
## +---+
## only showing top 3 rows

but it is:

• expensive - because it requires full shuffle and it something you typically want to avoid.
• suspicious - because order of values in a DataFrame is not something you can really depend on in non-trivial cases and since DataFrame doesn't support indexing it is relatively useless without collecting.

Comments

[Rahul Chawla]

Can you please elaborate DataFrame doesn't support indexing it is relatively useless without collecting.