How to spark-submit a python file in spark 2.1.0?
from pyspark.sql import SparkSession spark = SparkSession.builder.appName("appName").getOrCreate() sc = spark.sparkContext rdd = sc.parallelize([1,2,3,4,5,6,7]) print(rdd.count())
Run the above program with configurations you want : eg :
YOUR_SPARK_HOME/bin/spark-submit --master yourSparkMaster --num-executors 20 \ --executor-memory 1G --executor-cores 2 --driver-memory 1G \ pythonfile.py
These options are not mandatory. You can even run like
YOUR_SPARK_HOME/bin/spark-submit --master sparkMaster/local pythonfile.py
Kalyan 4 months
I am currently running spark 2.1.0. I have worked most of the time in PYSPARK shell, but I need to spark-submit a python file(similar to spark-submit jar in java) . How do you do that in python?