warning:Multiple versions of scala libraries detected?

13,979

Include the below in your pom

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_${scala.major.minor.version}</artifactId>
    <version>${spark.version}</version>
</dependency>
Share:
13,979
daxue
Author by

daxue

Updated on June 19, 2022

Comments

  • daxue
    daxue almost 2 years

    My project is maven+intellij. And I develop in windows system. firstly I use the lastest scala libray version 2.12.2.For getting the class SQLContext and so on,I have to import spark jar: enter image description here

    but then I was told that if I want to use this spark jar,I have to lower my scala version,So I delete the libiray and changed to 2.10....but now when I mvn_clean_install.I got this :

    [WARNING]  Expected all dependencies to require Scala version: 2.11.7
    [WARNING]  com.twitter:chill_2.11:0.5.0 requires scala version: 2.11.7
    [WARNING]  com.typesafe.akka:akka-remote_2.11:2.3.11 requires scala version: 2.11.7
    [WARNING]  com.typesafe.akka:akka-actor_2.11:2.3.11 requires scala version: 2.11.7
    [WARNING]  com.typesafe.akka:akka-slf4j_2.11:2.3.11 requires scala version: 2.11.7
    [WARNING]  org.apache.spark:spark-core_2.11:1.6.1 requires scala version: 2.11.7
    [WARNING]  org.json4s:json4s-jackson_2.11:3.2.10 requires scala version: 2.11.7
    [WARNING]  org.json4s:json4s-core_2.11:3.2.10 requires scala version: 2.11.7
    [WARNING]  org.json4s:json4s-ast_2.11:3.2.10 requires scala version: 2.11.7
    [WARNING]  org.json4s:json4s-core_2.11:3.2.10 requires scala version: 2.11.0
    [WARNING] Multiple versions of scala libraries detected!
    [INFO] E:\...\src\main\scala:-1: info: compiling
    [INFO] Compiling 4 source files to E:\...\target\classes at 1491813951772
    [ERROR] E:\...\qubole\mapreduce\ConvertToParquetFormat.scala:2: error: object sql is not a member of package org.apache.spark
    [ERROR] import org.apache.spark.sql.SQLContext
    [ERROR]                         ^
    [ERROR] E:\...\qubole\mapreduce\ConvertToParquetFormat.scala:15: error: not found: type SQLContext
    [ERROR]   val sqlContext = new SQLContext(sc)
    [ERROR]                        ^
    [ERROR] E:\...\mapreduce\ConvertToParquetFormat.scala:24: error: value toDF is not a member of org.apache.spark.rdd.RDD[....qubole.mapreduce.ConvertToParquetFormat.OmnitureHit]
    [ERROR] possible cause: maybe a semicolon is missing before `value toDF'?
    [ERROR]   .toDF().write.parquet ("file:///C:/Users/Desktop/456")
    [ERROR]    ^
    [ERROR] three errors found
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD FAILURE
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 03:00 min
    [INFO] Finished at: 2017-04-10T16:45:58+08:00
    [INFO] Final Memory: 33M/360M
    [INFO] ------------------------------------------------------------------------
    [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.0:compile (scala-compile-first) on project packages-omniture-qubole-mapreduce: wrap: org.apache.commons.exec.ExecuteException: Process exited with an er
    ror: 1 (Exit value: 1) -> [Help 1]
    [ERROR]
    [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
    [ERROR] Re-run Maven using the -X switch to enable full debug logging.
    [ERROR]
    [ERROR] For more information about the errors and possible solutions, please read the following articles:
    [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
    
    E:\github\mia\packages-omniture-qubole-mapreduce>
    

    so I delete the scala 10 version ,and it told me I do not have one scala libray.then I add a scala 2.11.since it looks like some jars need 2.11 version still Multiple versions of scala libraries detected?

    But when I ctrl+left_cilck the word SQLContext, I can go to the page ,but it is anonymous.

    enter image description here enter image description here Is that because I forget to delete something about the old jar or libary? and this is my left dependence list: enter image description hereenter image description hereenter image description here

    this is my pom.xml:

    <?xml version="1.0" encoding="UTF-8"?>
    <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
        <modelVersion>4.0.0</modelVersion>
    
        <groupId>com.company.www</groupId>
        <artifactId>packages-omniture-qubole-mapreduce</artifactId>
        <version>1.0-SNAPSHOT</version>
    
        <parent>
            <groupId>com.company.www.platform</groupId>
            <artifactId>platform-parent-spark</artifactId>
            <version>0.1.41</version>
        </parent>
    
        <properties>
            <spark.mapreduce.mainclass>com.company.www.packages.omniture.qubole.mapreduce.SampleMapReduceJob</spark.mapreduce.mainclass>
        </properties>
    
    
        <dependencies>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-core_${scala.major.minor.version}</artifactId>
                <version>${spark.version}</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>com.company.www.commons</groupId>
                <artifactId>commons-spark</artifactId>
                <version>[1.0.17, ]</version>
            </dependency>
            <dependency>
                <groupId>com.company.www</groupId>
                <artifactId>exp-user-interaction-messages-v1</artifactId>
                <version>[1.4,]</version>
            </dependency>
            <dependency>
                <groupId>org.scalaj</groupId>
                <artifactId>scalaj-http_${scala.major.minor.version}</artifactId>
                <version>1.1.4</version>
            </dependency>
            <dependency>
                <groupId>com.google.code.gson</groupId>
                <artifactId>gson</artifactId>
                <version>2.3</version>
            </dependency>
            <dependency>
                <groupId>org.parboiled</groupId>
                <artifactId>parboiled-java</artifactId>
                <version>1.0.2</version>
                <scope>test</scope>
            </dependency>
        </dependencies>
    
    
        <build>
            <plugins>
                <plugin>
                    <artifactId>maven-shade-plugin</artifactId>
                    <version>2.4</version>
                    <executions>
                        <execution>
                            <phase>package</phase>
                            <goals>
                                <goal>shade</goal>
                            </goals>
                            <configuration>
                                <finalName>packages-omniture-qubole-mapreduce</finalName>
                                <shadedArtifactAttached>false</shadedArtifactAttached>
                                <artifactSet>
                                    <includes>
                                        <include>*:*</include>
                                    </includes>
                                </artifactSet>
                                <filters>
                                    <filter>
                                        <artifact>*:*</artifact>
                                        <excludes>
                                            <exclude>META-INF/*.SF</exclude>
                                            <exclude>META-INF/*.DSA</exclude>
                                            <exclude>META-INF/*.RSA</exclude>
                                        </excludes>
                                    </filter>
                                </filters>
                                <transformers>
                                    <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                                    <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                        <resource>reference.conf</resource>
                                    </transformer>
                                    <transformer implementation="org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer">
                                        <resource>log4j.properties</resource>
                                    </transformer>
                                    <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                        <mainClass>${spark.mapreduce.mainclass}</mainClass>
                                    </transformer>
                                </transformers>
                                <relocations>
                                    <relocation>
                                        <pattern>org.eclipse.jetty</pattern>
                                        <shadedPattern>org.spark-project.jetty</shadedPattern>
                                        <includes>
                                            <include>org.eclipse.jetty.**</include>
                                        </includes>
                                    </relocation>
                                    <relocation>
                                        <pattern>com.google.common</pattern>
                                        <shadedPattern>org.spark-project.guava</shadedPattern>
                                        <excludes>
                                            <exclude>com/google/common/base/Absent*</exclude>
                                            <exclude>com/google/common/base/Function</exclude>
                                            <exclude>com/google/common/base/Optional*</exclude>
                                            <exclude>com/google/common/base/Present*</exclude>
                                            <exclude>com/google/common/base/Supplier</exclude>
                                        </excludes>
                                    </relocation>
                                </relocations>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
            </plugins>
        </build>
    
    </project>