How to spark-submit a python file in spark 2.1.0?

Question

I am currently running spark 2.1.0. I have worked most of the time in PYSPARK shell, but I need to spark-submit a python file(similar to spark-submit jar in java) . How do you do that in python?

Solution

pythonfile.py

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("appName").getOrCreate()
sc = spark.sparkContext
rdd = sc.parallelize([1,2,3,4,5,6,7])
print(rdd.count())

Run the above program with configurations you want : eg :

 YOUR_SPARK_HOME/bin/spark-submit --master yourSparkMaster --num-executors 20 \
        --executor-memory 1G --executor-cores 2 --driver-memory 1G \
        pythonfile.py

These options are not mandatory. You can even run like

YOUR_SPARK_HOME/bin/spark-submit --master sparkMaster/local pythonfile.py