Importing my jar to spark shell
13,409
Please make sure some below points it will works
1. start spark shell like ./spark-shell --jars jar_path
2. There is class file in jar under the same package which you import, open jar and check it.
3. After start spark go to http://localhost:4040/environment/ you jar will be in classpath entries or not.
Author by
Todor Markov
Updated on June 04, 2022Comments
-
Todor Markov almost 2 years
I have a simple scala maven module which is part of a larger project (I created it as described here: https://www.jetbrains.com/help/idea/2016.2/creating-a-maven-module.html):
package com.myorg.simplr import [...] @SerialVersionUID(100L) case class Simplr (){ //class code }
I am trying to use this class in spark shell, so I built a jar file "simplr-1.0.jar" and launched the spark shell with --jars simplr-1.0.jar.
Then, when I try to import, I get the following
scala> import com.myorg.simplr.Simplr <console>:25: error: object myorg is not a member of package com import com.myorg.simplr.Simplr ^
How can I make the import to work?
I used maven to build, and here's my pom.xml:
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <parent> <artifactId>my-parent-project</artifactId> <groupId>com.myorg</groupId> <version>1.0</version> </parent> <modelVersion>4.0.0</modelVersion> <artifactId>simplr</artifactId> <version>1.0</version> <dependencies> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.10</artifactId> <version>1.6.0</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-mllib_2.10</artifactId> <version>1.6.0</version> <scope>provided</scope> </dependency> </dependencies> </project>
-
Todor Markov over 7 yearsThanks, that helped. Apparently, IntelliJ didn't automatically add a build section to the pom.xml and maven wasn't building anything.