which jar contains org.apache.spark.sql.api.java.JavaSQLContext

Question

The following dependency is in the pom:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.10</artifactId>
    <version>1.3.0</version>
</dependency>

I expect the jar to contain the following class:

org.apache.spark.sql.api.java.JavaSQLContext

but while it contains the package org.apache.spark.sql.api.java, all that package appears to contain are interfaces named UDF1- UDSF22.

Which is the correct dependency to get JavaSQLContext?

Thanks.

Solution 1

The JavaSQLContext class has been removed from version 1.3.0 onwards. You should use org.apache.spark.sql.SQLContext class instead. The documentation states the following:

Prior to Spark 1.3 there were separate Java compatible classes (JavaSQLContext and JavaSchemaRDD) that mirrored the Scala API. In Spark 1.3 the Java API and Scala API have been unified. Users of either language should use SQLContext and DataFrame. In general theses classes try to use types that are usable from both languages (i.e. Array instead of language specific collections). In some cases where no common type exists (e.g., for passing in closures or Maps) function overloading is used instead.

Additionally the Java specific types API has been removed. Users of both Scala and Java should use the classes present in org.apache.spark.sql.types to describe schema programmatically.

As an aside if you want to search which jars contain a specific class you can use the Advanced Search of Maven Central and search "By Classname". So here is the search for JavaSQLContext: http://search.maven.org/#search|ga|1|fc%3A%22org.apache.spark.sql.api.java.JavaSQLContext%22

Solution 2

From a cursory search, it appears that the class org.apache.spark.sql.api.java.JavaSQLContext only appears in the 1.2 versions and earlier of the spark-sql JAR file. It is likely that the code with which you are working is also using this older dependency. You have two choices at this point: you can either upgrade your code usage, or you can downgrade the spark-sql JAR. You probably want to go with the former option.

If you insist on keeping your code the same, then including the following dependency in your POM should fix the problem:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.10</artifactId>
    <version>1.2.2</version>
</dependency>

If you want to upgrade your code, see the answer given by @DB5

Comments

[DB5]

Yes I did, but you didn't explain how the poster can upgrade their code to use the new code. I felt this was information worth adding. Didn't mean to cause any offence to you.

[Tim Biegeleisen]

No worries...but there are some SO copy cats out there with a nasty bite ^ ^