warning:Multiple versions of scala libraries detected?
Include the below in your pom
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.major.minor.version}</artifactId>
<version>${spark.version}</version>
</dependency>
daxue
Updated on June 19, 2022Comments
-
daxue almost 2 years
My project is maven+intellij. And I develop in windows system. firstly I use the lastest scala libray version 2.12.2.For getting the class SQLContext and so on,I have to import spark jar:
but then I was told that if I want to use this spark jar,I have to lower my scala version,So I delete the libiray and changed to 2.10....but now when I mvn_clean_install.I got this :
[WARNING] Expected all dependencies to require Scala version: 2.11.7 [WARNING] com.twitter:chill_2.11:0.5.0 requires scala version: 2.11.7 [WARNING] com.typesafe.akka:akka-remote_2.11:2.3.11 requires scala version: 2.11.7 [WARNING] com.typesafe.akka:akka-actor_2.11:2.3.11 requires scala version: 2.11.7 [WARNING] com.typesafe.akka:akka-slf4j_2.11:2.3.11 requires scala version: 2.11.7 [WARNING] org.apache.spark:spark-core_2.11:1.6.1 requires scala version: 2.11.7 [WARNING] org.json4s:json4s-jackson_2.11:3.2.10 requires scala version: 2.11.7 [WARNING] org.json4s:json4s-core_2.11:3.2.10 requires scala version: 2.11.7 [WARNING] org.json4s:json4s-ast_2.11:3.2.10 requires scala version: 2.11.7 [WARNING] org.json4s:json4s-core_2.11:3.2.10 requires scala version: 2.11.0 [WARNING] Multiple versions of scala libraries detected! [INFO] E:\...\src\main\scala:-1: info: compiling [INFO] Compiling 4 source files to E:\...\target\classes at 1491813951772 [ERROR] E:\...\qubole\mapreduce\ConvertToParquetFormat.scala:2: error: object sql is not a member of package org.apache.spark [ERROR] import org.apache.spark.sql.SQLContext [ERROR] ^ [ERROR] E:\...\qubole\mapreduce\ConvertToParquetFormat.scala:15: error: not found: type SQLContext [ERROR] val sqlContext = new SQLContext(sc) [ERROR] ^ [ERROR] E:\...\mapreduce\ConvertToParquetFormat.scala:24: error: value toDF is not a member of org.apache.spark.rdd.RDD[....qubole.mapreduce.ConvertToParquetFormat.OmnitureHit] [ERROR] possible cause: maybe a semicolon is missing before `value toDF'? [ERROR] .toDF().write.parquet ("file:///C:/Users/Desktop/456") [ERROR] ^ [ERROR] three errors found [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 03:00 min [INFO] Finished at: 2017-04-10T16:45:58+08:00 [INFO] Final Memory: 33M/360M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.0:compile (scala-compile-first) on project packages-omniture-qubole-mapreduce: wrap: org.apache.commons.exec.ExecuteException: Process exited with an er ror: 1 (Exit value: 1) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException E:\github\mia\packages-omniture-qubole-mapreduce>
so I delete the scala 10 version ,and it told me I do not have one scala libray.then I add a scala 2.11.since it looks like some jars need 2.11 version still
Multiple versions of scala libraries detected?
But when I ctrl+left_cilck the word SQLContext, I can go to the page ,but it is anonymous.
Is that because I forget to delete something about the old jar or libary? and this is my left dependence list:
this is my pom.xml:
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.company.www</groupId> <artifactId>packages-omniture-qubole-mapreduce</artifactId> <version>1.0-SNAPSHOT</version> <parent> <groupId>com.company.www.platform</groupId> <artifactId>platform-parent-spark</artifactId> <version>0.1.41</version> </parent> <properties> <spark.mapreduce.mainclass>com.company.www.packages.omniture.qubole.mapreduce.SampleMapReduceJob</spark.mapreduce.mainclass> </properties> <dependencies> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_${scala.major.minor.version}</artifactId> <version>${spark.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>com.company.www.commons</groupId> <artifactId>commons-spark</artifactId> <version>[1.0.17, ]</version> </dependency> <dependency> <groupId>com.company.www</groupId> <artifactId>exp-user-interaction-messages-v1</artifactId> <version>[1.4,]</version> </dependency> <dependency> <groupId>org.scalaj</groupId> <artifactId>scalaj-http_${scala.major.minor.version}</artifactId> <version>1.1.4</version> </dependency> <dependency> <groupId>com.google.code.gson</groupId> <artifactId>gson</artifactId> <version>2.3</version> </dependency> <dependency> <groupId>org.parboiled</groupId> <artifactId>parboiled-java</artifactId> <version>1.0.2</version> <scope>test</scope> </dependency> </dependencies> <build> <plugins> <plugin> <artifactId>maven-shade-plugin</artifactId> <version>2.4</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <finalName>packages-omniture-qubole-mapreduce</finalName> <shadedArtifactAttached>false</shadedArtifactAttached> <artifactSet> <includes> <include>*:*</include> </includes> </artifactSet> <filters> <filter> <artifact>*:*</artifact> <excludes> <exclude>META-INF/*.SF</exclude> <exclude>META-INF/*.DSA</exclude> <exclude>META-INF/*.RSA</exclude> </excludes> </filter> </filters> <transformers> <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" /> <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"> <resource>reference.conf</resource> </transformer> <transformer implementation="org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer"> <resource>log4j.properties</resource> </transformer> <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> <mainClass>${spark.mapreduce.mainclass}</mainClass> </transformer> </transformers> <relocations> <relocation> <pattern>org.eclipse.jetty</pattern> <shadedPattern>org.spark-project.jetty</shadedPattern> <includes> <include>org.eclipse.jetty.**</include> </includes> </relocation> <relocation> <pattern>com.google.common</pattern> <shadedPattern>org.spark-project.guava</shadedPattern> <excludes> <exclude>com/google/common/base/Absent*</exclude> <exclude>com/google/common/base/Function</exclude> <exclude>com/google/common/base/Optional*</exclude> <exclude>com/google/common/base/Present*</exclude> <exclude>com/google/common/base/Supplier</exclude> </excludes> </relocation> </relocations> </configuration> </execution> </executions> </plugin> </plugins> </build> </project>