SBT Assembly - Deduplicate error & Exclude error
Try this one to exclude only ECLIPSEF.RSA
file, because excluding whole dependencies, which your libraries may need, is a bad idea :)
Deduplication error in build.sbt when building a fat jar using sbt assembly
Solution for latest 0.12.0
version of assembly plugin:
assemblyMergeStrategy in assembly := {
case PathList(ps @ _*) if ps.last endsWith ".RSA" => MergeStrategy.first
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
project/assembly.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")
See https://github.com/sbt/sbt-assembly#merge-strategy
P.S. About exclude
: you can't just specify exclude
in libraryDependencies - you need something like this: "log4j" % "log4j" % "1.2.15" exclude("javax.jms", "jms")
. See Library Management
macha
Updated on July 27, 2022Comments
-
macha almost 2 years
I am trying to build a JAR with dependencies using sbt-assembly. But I am running into this error again and again. I have tried multiple different things but I end up here. I am pretty new to SBT and wanted to get some help on this one. Here are the build.sbt & assembly.sbt files.
build.sbt
seq(assemblySettings: _*) name := "StreamTest" version := "1.0" scalaVersion := "2.10.4" libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.0" libraryDependencies += "org.apache.spark" % "spark-streaming-kinesis-asl_2.10" % "1.1.0"
project/assembly.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")
When I run the sbt assembly command, I am getting this error below.
[info] Including: joda-time-2.5.jar [info] Checking every *.class/*.jar file's SHA-1. [info] Merging files... [warn] Merging 'about_files/LICENSE.txt' with strategy 'rename' [warn] Merging 'about_files/NOTICE.txt' with strategy 'rename' [warn] Merging 'META-INF/NOTICE.txt' with strategy 'rename' [warn] Merging 'META-INF/NOTICE' with strategy 'rename' [warn] Merging 'org/xerial/snappy/native/README' with strategy 'rename' [warn] Merging 'META-INF/maven/org.xerial.snappy/snappy-java/LICENSE' with strategy 'rename' [warn] Merging 'META-INF/license' with strategy 'rename' [warn] Merging 'about.html' with strategy 'rename' [warn] Merging 'META-INF/LICENSE.txt' with strategy 'rename' [warn] Merging 'META-INF/README.txt' with strategy 'rename' [warn] Merging 'LICENSE.txt' with strategy 'rename' [warn] Merging 'META-INF/LICENSE' with strategy 'rename' [warn] Merging 'META-INF/DEPENDENCIES' with strategy 'discard' java.lang.RuntimeException: deduplicate: different file contents found in the following: /Users/user/.ivy2/cache/org.eclipse.jetty.orbit/javax.transaction/orbits/javax.transaction-1.1.1.v201105210645.jar:META-INF/ECLIPSEF.RSA /Users/user/.ivy2/cache/org.eclipse.jetty.orbit/javax.servlet/orbits/javax.servlet-3.0.0.v201112011016.jar:META-INF/ECLIPSEF.RSA /Users/user/.ivy2/cache/org.eclipse.jetty.orbit/javax.mail.glassfish/orbits/javax.mail.glassfish-1.4.1.v201005082020.jar:META-INF/ECLIPSEF.RSA /Users/user/.ivy2/cache/org.eclipse.jetty.orbit/javax.activation/orbits/javax.activation-1.1.0.v201105071233.jar:META-INF/ECLIPSEF.RSA at sbtassembly.Plugin$Assembly$.sbtassembly$Plugin$Assembly$$applyStrategy$1(Plugin.scala:253) at sbtassembly.Plugin$Assembly$$anonfun$15.apply(Plugin.scala:270) at sbtassembly.Plugin$Assembly$$anonfun$15.apply(Plugin.scala:267) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) at sbtassembly.Plugin$Assembly$.applyStrategies(Plugin.scala:272) at sbtassembly.Plugin$Assembly$.x$4$lzycompute$1(Plugin.scala:172) at sbtassembly.Plugin$Assembly$.x$4$1(Plugin.scala:170) at sbtassembly.Plugin$Assembly$.stratMapping$lzycompute$1(Plugin.scala:170) at sbtassembly.Plugin$Assembly$.stratMapping$1(Plugin.scala:170) at sbtassembly.Plugin$Assembly$.inputs$lzycompute$1(Plugin.scala:214) at sbtassembly.Plugin$Assembly$.inputs$1(Plugin.scala:204) at sbtassembly.Plugin$Assembly$.apply(Plugin.scala:230) at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:373) at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:370) at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47) at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40) at sbt.std.Transform$$anon$4.work(System.scala:63) at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226) at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226) at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17) at sbt.Execute.work(Execute.scala:235) at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226) at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226) at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159) at sbt.CompletionService$$anon$2.call(CompletionService.scala:28) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) [error] (*:assembly) deduplicate: different file contents found in the following: [error] /Users/user/.ivy2/cache/org.eclipse.jetty.orbit/javax.transaction/orbits/javax.transaction-1.1.1.v201105210645.jar:META-INF/ECLIPSEF.RSA [error] /Users/user/.ivy2/cache/org.eclipse.jetty.orbit/javax.servlet/orbits/javax.servlet-3.0.0.v201112011016.jar:META-INF/ECLIPSEF.RSA [error] /Users/user/.ivy2/cache/org.eclipse.jetty.orbit/javax.mail.glassfish/orbits/javax.mail.glassfish-1.4.1.v201005082020.jar:META-INF/ECLIPSEF.RSA [error] /Users/user/.ivy2/cache/org.eclipse.jetty.orbit/javax.activation/orbits/javax.activation-1.1.0.v201105071233.jar:META-INF/ECLIPSEF.RSA [error] Total time: 23 s, completed Nov 28, 2014 9:32:53 PM
sbt-version
0.13.6
EDIT
Now after looking around I have made another change to exclude any dependencies, as part of another question on stackoverflow.
Updated build.sbt
seq(assemblySettings: _*) name := "StreamTest" version := "1.0" scalaVersion := "2.10.4" libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.0" libraryDependencies += "org.apache.spark" % "spark-streaming-kinesis-asl_2.10" % "1.1.0" libraryDependencies ++= Seq( exclude("org.eclipse.jetty.orbit", "javax.servlet"). exclude("org.eclipse.jetty.orbit", "javax.transaction"). exclude("org.eclipse.jetty.orbit", "javax.mail"). exclude("org.eclipse.jetty.orbit", "javax.activation"). exclude("commons-beanutils", "commons-beanutils-core"). exclude("commons-collections", "commons-collections"). exclude("commons-collections", "commons-collections"). exclude("com.esotericsoftware.minlog", "minlog") )
When I run the assembly command again, this is the error that I get.
build.sbt:14: error: not found: value exclude exclude("org.eclipse.jetty.orbit", "javax.servlet"). ^
EDIT 2:
Updated build.sbt
import AssemblyKeys._ seq(assemblySettings: _*) name := "SparkStreamingKinesis" version := "1.0" scalaVersion := "2.10.4" libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.0" libraryDependencies += "org.apache.spark" % "spark-streaming-kinesis-asl_2.10" % "1.1.0" assemblyMergeStrategy in assembly := { case PathList(ps @ _*) if ps.last endsWith ".RSA" => MergeStrategy.first case x => val oldStrategy = (assemblyMergeStrategy in assembly).value oldStrategy(x) }
After making the update, here is the error that I am getting. I tried to do an import of assemblyMergeStrategy, but it doesn't look like a class that I can import.
build.sbt:21: error: not found: value assemblyMergeStrategy assemblyMergeStrategy in assembly := { ^ [error] Type error in expression
-
macha over 9 yearsthank you for the quick reply dk14. When I added the assemblyMergeStrategy, I ran into another error. I have updated the question.
-
dk14 over 9 yearsjust add
import AssemblyKeys._
-
macha over 9 yearshey dk14, I added it to the build.sbt file but I still get the same error.
-
dk14 over 9 yearsMove to the latest version of plugin: 0.12.0 + sbt 0.13.6+. If you have older version of sbt - check this answer stackoverflow.com/questions/25925233/…