How to pass external parameters through Spark submit

Question

In my Application, i need to connect to the database so i need to pass IP address and database name when application is submitted.

I submit the application as follows: :

./spark-submit --class class name --master spark://localhost:7077 \
--deploy-mode client /home/hadoop/myjar.jar

Solution 1

If you check the official documentation you'll see that spark-submit has following syntax:

./bin/spark-submit \
  --class <main-class>
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  --conf <key>=<value> \
  ... # other options
  <application-jar> \
  [application-arguments]

You can use either application-arguments and conf to pass required configuration to the main method and SparkConf respectively.

Solution 2

As stated by zero323 you can use the spark-submit command from the link

  ./bin/spark-submit \
  --class <main-class>
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  --conf <key>=<value> \
  ... # other options
  <application-jar> \
  [application-arguments]

Here, conf is used to pass the Spark related configs which are required for the application to run like any specific property(executor memory) or if you want to override the default property which is set in Spark-default.conf.

As far as your use case is concerned you want to pass the IP to the application to connect to database then you can use the [application-arguments] which are passed after the JAR.

When you set up your main as:

def main(args: Array[String])

Then you can accept anything as an argument given after .jar line.

Please refer for more details