sbt-assembly: deduplication found error

Question

I am not sure whether mergestrategy or exclude jars is the best option here. Any help with how do I proceed further with this error will be great!

[sameert@pzxdcc0151 approxstrmatch]$ sbt assembly
[info] Loading project definition from /apps/sameert/software/approxstrmatch/project
[info] Set current project to approxstrmatch (in build file:/apps/sameert/software/approxstrmatch/)
[info] Including from cache: scala-library.jar
[info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[info] Including from cache: curator-client-2.4.0.jar
[info] Including from cache: secondstring-20140729.jar
[info] Including from cache: slf4j-api-1.7.5.jar
[info] Including from cache: jsr305-1.3.9.jar
[info] Including from cache: jul-to-slf4j-1.7.5.jar
[info] Including from cache: jcl-over-slf4j-1.7.5.jar
[info] Including from cache: commons-digester-1.8.jar
[info] Including from cache: compress-lzf-1.0.0.jar
[info] Including from cache: commons-beanutils-1.7.0.jar
[info] Including from cache: zookeeper-3.4.5.jar
[info] Including from cache: slf4j-log4j12-1.7.5.jar
[info] Including from cache: commons-beanutils-core-1.8.0.jar
[info] Including from cache: commons-net-2.2.jar
[info] Including from cache: commons-el-1.0.jar
[info] Including from cache: log4j-1.2.17.jar
[info] Including from cache: scala-library.jar
[info] Including from cache: jline-0.9.94.jar
[info] Including from cache: snappy-java-1.0.5.jar
[info] Including from cache: hsqldb-1.8.0.10.jar
[info] Including from cache: chill_2.10-0.3.6.jar
[info] Including from cache: oro-2.0.8.jar
[info] Including from cache: chill-java-0.3.6.jar
[info] Including from cache: kryo-2.21.jar
[info] Including from cache: reflectasm-1.07-shaded.jar
[info] Including from cache: minlog-1.2.jar
[info] Including from cache: guava-14.0.1.jar
[info] Including from cache: jetty-plus-8.1.14.v20131031.jar
[info] Including from cache: javax.transaction-1.1.1.v201105210645.jar
[info] Including from cache: jackson-mapper-asl-1.8.8.jar
[info] Including from cache: jackson-core-asl-1.8.8.jar
[info] Including from cache: jetty-webapp-8.1.14.v20131031.jar
[info] Including from cache: curator-recipes-2.4.0.jar
[info] Including from cache: jetty-xml-8.1.14.v20131031.jar
[info] Including from cache: spark-core_2.10-1.0.0.jar
[info] Including from cache: objenesis-1.2.jar
[info] Including from cache: curator-framework-2.4.0.jar
[info] Including from cache: hadoop-client-1.0.4.jar
[info] Including from cache: jetty-util-8.1.14.v20131031.jar
[info] Including from cache: scalap-2.10.4.jar
[info] Including from cache: akka-remote_2.10-2.2.3-shaded-protobuf.jar
[info] Including from cache: jetty-servlet-8.1.14.v20131031.jar
[info] Including from cache: jetty-security-8.1.14.v20131031.jar
[info] Including from cache: jetty-server-8.1.14.v20131031.jar
[info] Including from cache: javax.servlet-3.0.0.v201112011016.jar
[info] Including from cache: jetty-continuation-8.1.14.v20131031.jar
[info] Including from cache: jetty-http-8.1.14.v20131031.jar
[info] Including from cache: jetty-io-8.1.14.v20131031.jar
[info] Including from cache: hadoop-core-1.0.4.jar
[info] Including from cache: jetty-jndi-8.1.14.v20131031.jar
[info] Including from cache: xmlenc-0.52.jar
[info] Including from cache: commons-codec-1.4.jar
[info] Including from cache: javax.mail.glassfish-1.4.1.v201005082020.jar
[info] Including from cache: javax.activation-1.1.0.v201105071233.jar
[info] Including from cache: commons-math-2.1.jar
[info] Including from cache: commons-lang3-3.3.2.jar
[info] Including from cache: commons-configuration-1.6.jar
[info] Including from cache: metrics-core-3.0.0.jar
[info] Including from cache: metrics-jvm-3.0.0.jar
[info] Including from cache: metrics-json-3.0.0.jar
[info] Including from cache: commons-collections-3.2.1.jar
[info] Including from cache: metrics-graphite-3.0.0.jar
[info] Including from cache: commons-lang-2.4.jar
[info] Including from cache: akka-actor_2.10-2.2.3-shaded-protobuf.jar
[info] Including from cache: config-1.0.2.jar
[info] Including from cache: tachyon-0.4.1-thrift.jar
[info] Including from cache: netty-3.6.6.Final.jar
[info] Including from cache: protobuf-java-2.4.1-shaded.jar
[info] Including from cache: uncommons-maths-1.2.2a.jar
[info] Including from cache: akka-slf4j_2.10-2.2.3-shaded-protobuf.jar
[info] Including from cache: json4s-jackson_2.10-3.2.6.jar
[info] Including from cache: json4s-core_2.10-3.2.6.jar
[info] Including from cache: ant-1.9.0.jar
[info] Including from cache: json4s-ast_2.10-3.2.6.jar
[info] Including from cache: ant-launcher-1.9.0.jar
[info] Including from cache: paranamer-2.6.jar
[info] Including from cache: commons-io-2.4.jar
[info] Including from cache: jackson-core-2.3.0.jar
[info] Including from cache: pyrolite-2.0.1.jar
[info] Including from cache: colt-1.2.0.jar
[info] Including from cache: concurrent-1.3.4.jar
[info] Including from cache: py4j-0.8.1.jar
[info] Including from cache: mesos-0.18.1-shaded-protobuf.jar
[info] Including from cache: scala-compiler.jar
[info] Including from cache: jets3t-0.7.1.jar
[info] Including from cache: commons-httpclient-3.1.jar
[info] Including from cache: netty-all-4.0.17.Final.jar
[info] Including from cache: stream-2.5.1.jar
[info] Including from cache: scala-reflect.jar
[info] Including from cache: jackson-databind-2.3.0.jar
[info] Including from cache: jackson-annotations-2.3.0.jar
[warn] Merging 'META-INF/MANIFEST.MF' with strategy 'discard'
[warn] Strategy 'discard' was applied to a file
[info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[warn] Merging 'META-INF/DEPENDENCIES' with strategy 'discard'
[info] Assembly up to date: /apps/sameert/software/approxstrmatch/app/target/scala-2.10/app-assembly-0.1-SNAPSHOT.jar

// Here is where I start seeing errors:

java.lang.RuntimeException: deduplicate: different file contents found in the following:
/home/sameert/.ivy2/cache/org.eclipse.jetty.orbit/javax.transaction/orbits/javax.transaction-1.1.1.v201105210645.jar:META-INF/ECLIPSEF.RSA
/home/sameert/.ivy2/cache/org.eclipse.jetty.orbit/javax.servlet/orbits/javax.servlet-3.0.0.v201112011016.jar:META-INF/ECLIPSEF.RSA
/home/sameert/.ivy2/cache/org.eclipse.jetty.orbit/javax.mail.glassfish/orbits/javax.mail.glassfish-1.4.1.v201005082020.jar:META-INF/ECLIPSEF.RSA
/home/sameert/.ivy2/cache/org.eclipse.jetty.orbit/javax.activation/orbits/javax.activation-1.1.0.v201105071233.jar:META-INF/ECLIPSEF.RSA
    at sbtassembly.Plugin$Assembly$.sbtassembly$Plugin$Assembly$$applyStrategy$1(Plugin.scala:253)
    at sbtassembly.Plugin$Assembly$$anonfun$15.apply(Plugin.scala:270)
    at sbtassembly.Plugin$Assembly$$anonfun$15.apply(Plugin.scala:267)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
    at sbtassembly.Plugin$Assembly$.applyStrategies(Plugin.scala:272)
    at sbtassembly.Plugin$Assembly$.x$4$lzycompute$1(Plugin.scala:172)
    at sbtassembly.Plugin$Assembly$.x$4$1(Plugin.scala:170)
    at sbtassembly.Plugin$Assembly$.stratMapping$lzycompute$1(Plugin.scala:170)
    at sbtassembly.Plugin$Assembly$.stratMapping$1(Plugin.scala:170)
    at sbtassembly.Plugin$Assembly$.inputs$lzycompute$1(Plugin.scala:214)
    at sbtassembly.Plugin$Assembly$.inputs$1(Plugin.scala:204)
    at sbtassembly.Plugin$Assembly$.apply(Plugin.scala:230)
    at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:373)
    at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:370)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42)
    at sbt.std.Transform$$anon$4.work(System.scala:64)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
    at sbt.Execute.work(Execute.scala:244)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

// Here are the error messages.

[error] (approxstrmatch/*:assembly) deduplicate: different file contents found in the following:
[error] /home/sameert/.ivy2/cache/org.eclipse.jetty.orbit/javax.transaction/orbits/javax.transaction-1.1.1.v201105210645.jar:META-INF/ECLIPSEF.RSA
[error] /home/sameert/.ivy2/cache/org.eclipse.jetty.orbit/javax.servlet/orbits/javax.servlet-3.0.0.v201112011016.jar:META-INF/ECLIPSEF.RSA
[error] /home/sameert/.ivy2/cache/org.eclipse.jetty.orbit/javax.mail.glassfish/orbits/javax.mail.glassfish-1.4.1.v201005082020.jar:META-INF/ECLIPSEF.RSA
[error] /home/sameert/.ivy2/cache/org.eclipse.jetty.orbit/javax.activation/orbits/javax.activation-1.1.0.v201105071233.jar:META-INF/ECLIPSEF.RSA
[error] Total time: 4 s, completed Aug 5, 2014 9:53:06 AM

Solution 1

Add the code below to your build.sbt file

assemblyMergeStrategy in assembly := {
 case PathList("META-INF", xs @ _*) => MergeStrategy.discard
 case x => MergeStrategy.first
}

This helped me a lot.

Solution 2

Use the "provided" configuration, which will scope your dependent library.

For example:

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.0" % "provided"

If needed, read more at

https://github.com/sbt/sbt-assembly#excluding-jars-and-files

Solution 3

import AssemblyKeys._

name := "approxstrmatch"

version := "1.0"

scalaVersion := "2.10.4"

// unmanagedJars in Compile += file("lib/secondstring-20140729.jar")

libraryDependencies+="org.apache.spark"%%"spark-core"%"1.0.0"

libraryDependencies ++= Seq(
    ("org.apache.spark"%%"spark-core"%"1.0.0").
    exclude("org.eclipse.jetty.orbit", "javax.servlet").
    exclude("org.eclipse.jetty.orbit", "javax.transaction").
    exclude("org.eclipse.jetty.orbit", "javax.mail").
     exclude("org.eclipse.jetty.orbit", "javax.activation").
    exclude("commons-beanutils", "commons-beanutils-core").
    exclude("commons-collections", "commons-collections").
    exclude("commons-collections", "commons-collections").
    exclude("com.esotericsoftware.minlog", "minlog")
)


resolvers += "AkkaRepository" at "http://repo.akka.io/releases/"



 lazy val app = Project("approxstrmatch", file("approxstrmatch"),
    settings = buildSettings ++ assemblySettings ++ Seq(
    mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
    {
        case PathList("javax", "servlet", xs @ _*)         => MergeStrategy.first
        case PathList("javax", "transaction", xs @ _*)     => MergeStrategy.first
        case PathList("javax", "mail", xs @ _*)     => MergeStrategy.first
        case PathList("javax", "activation", xs @ _*)     => MergeStrategy.first
        case PathList(ps @ _*) if ps.last endsWith ".html" => MergeStrategy.first
        case "application.conf" => MergeStrategy.concat
        case "unwanted.txt"     => MergeStrategy.discard
        case x => old(x)
        }
    })
  )


mainClass in assembly := Some("approxstrmatch.JaccardScore")
// jarName in assembly := "approstrmatch.jar"

Solution 4

The module-info.class file has moved in many libraries. Here is the updated solution

ThisBuild / assemblyMergeStrategy  := {
  case PathList("module-info.class") => MergeStrategy.discard
  case x if x.endsWith("/module-info.class") => MergeStrategy.discard
  case x =>
    val oldStrategy = (ThisBuild / assemblyMergeStrategy).value
    oldStrategy(x)
}

View more solutions

Comments

[samthebest]

Sorry, I had a "skip update" in a script, DOH. This did work and is significantly simpler than the other answer.

[HappyCoding]

using "provided" works for me as well. it will exclude that provided dependencies and keep the jar package light. However, if the I want to pack all the dependencies/jars, how to do that?

[note]

Actually I think we should just overwrite merge strategy for META-INF leaving old strategies for the rest, so: assemblyMergeStrategy in assembly := { case PathList("META-INF", xs @ _*) => MergeStrategy.discard\n case x => val oldStrategy = (assemblyMergeStrategy in assembly).value oldStrategy(x) }

[Minh Thai]

You can refer to sbt-assembly/Merge Strategy for more details

[thebluephantom]

Worked for me, never heard anyone about this.

[Maldus]

Works well, but a small explaination on what it does wouldn't hurt

[Tushar Walzade]

It was not working as it is in Play 2.5; So, I replaced assemblyMergeStrategy by mergeStrategy & it worked!

[David Boreham]

@TusharWalzade mergeStrategy is also working for me.

[ianpojman]

The reason this fixes is what you are telling SBT's assembly plugin to do is to merge a bunch of jar files. What do you do if multiple jar files have a file at the same path? And what if they are different? Adding this rule to default to just picking the first will usually get your build to work, but it doesn't do this by default because this could cause a situation where the "Wrong" file is chosen and included in the fat JAR. Hence SBT requires you to explicitly define how you want to handle conflicts.