how to submit spark application on cmd

Question

I have a Spark Application which I initially created using maven (on windows). I converted my maven project into an Eclipse project, and I am now working on it via Eclipse.
I am able to utilize and verify this by running the application through Eclipse.

Now, I am trying to submit my application through the command line. I use the spark-submit command as follows:

bin\spark-submit \  
  --class C:\Users\pc\Desktop\eclipse\myapp\src\main\java\myapp\Main.java \  
  --master local[8] \
  C:\Users\pc\Desktop\eclipse\myjar.jar

When I submit my jar, I get an error saying can't load main class from Jar file?

what i am doing wrong?
is it the path of my main class (--class C:\Users\pc\Desktop\eclipse\myapp\src\main\java\myapp\Main.java)

Solution 1

--class: The entry point for your application (e.g. org.apache.spark.examples.SparkPi)

This is not the path to the file on your local hard drive. Think about a distributed cluster of computers. They do not all have access to the file path you have given.

Looks like you should use myapp.Main

Same goes for the JAR file.

application-jar: Path to a bundled jar including your application and all dependencies. The URL must be globally visible inside of your cluster, for instance, an hdfs:// path or a file:// path that is present on all nodes.

Solution 2

i was using uneeded sleshes that s why i couldn't deploy it correctly on windows, this is the right way to submit it

bin\spark-submit --class myapp.Main -- master local[*]       file:///C:\Users\pc\Desktop\eclipse\myjar.jar

Comments

[hammadspark]

how it should look the path to my jar? would you give me an example please?

[OneCricketeer]

If you have a standalone cluster, then what you have should be fine. If you had a distributed cluster of Linux boxes, then say your JAR was on each machine at /tmp/myapp.jar, then you would say file:///tmp/myapp.jar, if you have some distributed server like HDFS or S3, replace file:// with hdfs:// or s3:// there and point to the path of the file.

[hammadspark]

now i am using a local mode just for testing but i ll use it with hdfs later, i ll try this and keep you informed. Thank you

[OneCricketeer]

What path to the jar file did you use?

[giaosudau]

How do you build the jar? Show me the full log and the struct of your project folder and maven or sbt file.

[hammadspark]

this s the path C:\Users\pc\Desktop\eclipse\myjar.jar

[hammadspark]

this s the struct of my project: myapp->src->main->java->myapp->Main the package myapp contais the main class Main

[hammadspark]

i was using maven but i got an exception (here is a link for more detail stackoverflow.com/questions/37848264/…), so i created the jar manually (export->java->Runnable jar file)

[OneCricketeer]

Try file:///C:/Users/pc/Desktop/eclipse/myjar.jar

[hammadspark]

this is exactly what is written in the cmd :

[hammadspark]

C:\Users\pc\Desktop\spark-1.2.1-bin-hadoop2.4>./bin/spark-su‌​bmit \--class myapp.Main\ --master local[*] \ file:///C:/Users/pc/Desktop/eclipse/myjar.jar

[hammadspark]

but this time it doesn't accept the command because it starts with '.'

[OneCricketeer]

Looks okay... Are you sure the jar file is being packaged correctly? You can open it a program that opens ZIP files. As for the '.', you are confusing a Unix shell with a CMD. The slashes go the opposite direction in Windows

[hammadspark]

i created the jar fie manually bcause i have a mojoExecutionException with maven, but when i click on it doesn't start

[OneCricketeer]

I wouldn't trust building a JAR manually if you have Maven. You should let mvn package build it.

[hammadspark]

i tried with slashes to the opposite direction but it s always the same error(can not load class main) i think i should fix the problem with maven and rebuild it again

[hammadspark]

would you check this error please? stackoverflow.com/questions/37848264/…

[OneCricketeer]

Seems like a good plan. You can refer here, if needed for a Spark pom.xml config robertomarchetto.com/spark_java_maven_example