How to resolve a library conflict (apache commons-codec)

android apache conflict nosuchmethoderror apache-commons-codec

11,958

Solution 1

Late reply but maybe usefull for someone.

Problem solved by using Maven Shade Plugin

This plugin allows to rename package names of conflicted library at compilation.

Usage :

   <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <executions>
            <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
                </goals>
            <configuration>
                    <relocations>
                        <relocation>
                                <pattern>org.apache.commons</pattern>
                                    <shadedPattern>com.example.shaded.org.apache.commons</shadedPattern>
                        </relocation>
                    </relocations>
                        <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
            </configuration>
            </execution>
        </executions>
    </plugin>

Solution 2

Expanded version of nbe_42's answer, with full documentation...

The commons-codec-shaded jar project:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>commons-codec</groupId>
    <artifactId>commons-codec-shaded</artifactId>
    <name>Apache Commons Codec (shaded)</name>
    <!-- The version of this project specifies the Apache Commons Codec version which will
         be used, it must therefore match an existing (and preferably current) version. -->
    <version>1.9</version>
    <packaging>jar</packaging>

    <!--
      *************************************************************
       Rationale for this "shaded" version of Apache Commons Codec
      *************************************************************

      Context:
        Android includes an outdated version (v1.3) of commons-codec as an internal library.
        This library is not exposed in the Android SDK so app developers who want to rely on
        commons-codec need to treat it as an addition dependency and include it in the APK
        of their app. However, at runtime Android will always favour its internal version of
        the library which causes trouble when app code tries to call methods that don't
        exist in v1.3 but do exist in the version the developer expected to be using.

      Solution:
        After experimenting with many different variations the current (and final) solution
        to this problem is implemented in this project and does not require big hacks or
        changes in projects which depend on commons-codec, expect for declaring dependency
        on commons-codec-shaded (i.e. this project) instead of the original commons-codec.
        What we do here is take the "original" commons-codec library (currently version 1.9)
        and use the maven-shade-plugin to "shade" it, which means we modify the package name
        of the library (both in the compiled classes and the sources jar) in order to avoid
        the clash with Android's version. The package name is changes from
        "org.apache.commons.codec" to "shaded.org.apache.commons.codec". The result is
        published to the local Maven repository for other projects to use by simple
        dependency declaration on this project. Because we only apply the shading to
        commons-codec itself (and not to other classes using it; which is possible using the
        shade plug-in but doesn't work in combination with android-maven-plugin) any client
        classes which make use of commons-codec will have to import the new "shaded" package
        name instead of the old one.

      Issue on android-maven-plugin github which I posted to discuss all this:
        https://github.com/jayway/maven-android-plugin/issues/487
    -->

    <description>
     The Apache Commons Codec package contains simple encoder and decoders for
     various formats such as Base64 and Hexadecimal.  In addition to these
     widely used encoders and decoders, the codec package also maintains a
     collection of phonetic encoding utilities.
    </description>
    <url>http://commons.apache.org/proper/commons-codec/</url>
    <organization>
        <name>The Apache Software Foundation</name>
        <url>http://www.apache.org/</url>
    </organization>
    <licenses>
        <license>
            <name>The Apache Software License, Version 2.0</name>
            <url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
            <distribution>repo</distribution>
        </license>
    </licenses>

    <contributors>
        <contributor>
            <name>Matthias Stevens</name>
            <email>m.stevens {at} ucl.ac.uk</email>
            <roles>
                <role>Shading for use on Android</role>
            </roles>
        </contributor>
        <!-- see commons-codec:commons-codec pom for original contributors/developers -->
    </contributors>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <maven.compiler.source>1.6</maven.compiler.source>
        <maven.compiler.target>1.6</maven.compiler.target>
        <commons-codec-package>org.apache.commons.codec</commons-codec-package>
        <shading.prefix>shaded</shading.prefix>
        <shaded-commons-codec-package>${shading.prefix}.${commons-codec-package}</shaded-commons-codec-package>
        <commons-codec-src-folder>${project.build.directory}/commons-codec-src</commons-codec-src-folder>
        <commons-codec-res-folder>${project.build.directory}/commons-codec-res</commons-codec-res-folder>
        <manifest.path>${project.build.directory}/MANIFEST.MF</manifest.path>
        <!-- plugin versions -->
        <dependency-plugin-version>2.9</dependency-plugin-version>
        <compiler-plugin-version>3.2</compiler-plugin-version>
        <antrun-plugin-version>1.7</antrun-plugin-version>
        <jar-plugin-version>2.5</jar-plugin-version>
        <source-plugin-version>2.4</source-plugin-version>
        <shade-plugin-version>2.3</shade-plugin-version>
        <bundle-plugin-version>2.5.3</bundle-plugin-version>
        <!-- taken/modified from: http://svn.apache.org/repos/asf/commons/proper/commons-parent/trunk/pom.xml -->
        <commons.osgi.symbolicName>${shaded-commons-codec-package}</commons.osgi.symbolicName>
        <commons.osgi.export>${shaded-commons-codec-package}.*;version=${project.version};-noimport:=true</commons.osgi.export>
        <commons.osgi.import>*</commons.osgi.import>
        <commons.osgi.dynamicImport />
        <commons.osgi.private />
    </properties>

    <build>
        <finalName>${project.artifactId}</finalName>
        <sourceDirectory>${commons-codec-src-folder}</sourceDirectory>      
        <resources>
            <resource>
                <!-- txt files in shaded\org\apache\commons\codec\language\bm -->
                <directory>${commons-codec-res-folder}</directory>
                <includes>
                    <include>${shading.prefix}/**/*.txt</include>
                </includes>
            </resource>
            <resource>
                <!-- LICENSE & NOTICE files -->
                <directory>${commons-codec-res-folder}/META-INF</directory>
                <targetPath>META-INF</targetPath>
                <includes>
                    <include>*.txt</include>
                </includes>
            </resource>
        </resources>
        <plugins>
            <plugin>
                <!-- fetch & unpack commons-codec sources and resources -->
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>${dependency-plugin-version}</version>
                <executions>
                    <execution>
                        <id>unpack_commons-codec_sources_and_resources</id>
                        <phase>process-sources</phase>
                        <goals>
                            <goal>unpack</goal>
                        </goals>
                        <configuration>
                            <artifactItems>
                                <!-- commons-codec sources -->
                                <artifactItem>
                                    <groupId>commons-codec</groupId>
                                    <artifactId>commons-codec</artifactId>
                                    <!-- the project version specifies the commons-codec version to use: -->
                                    <version>${project.version}</version>
                                    <classifier>sources</classifier>
                                    <overWrite>true</overWrite>
                                    <excludes>**/*.txt,META-INF/*</excludes>
                                    <outputDirectory>${commons-codec-src-folder}</outputDirectory>
                                </artifactItem>
                                <!-- commons-codec resources (in package) -->
                                <artifactItem>
                                    <groupId>commons-codec</groupId>
                                    <artifactId>commons-codec</artifactId>
                                    <!-- the project version specifies the commons-codec version to use: -->
                                    <version>${project.version}</version>
                                    <classifier>sources</classifier>
                                    <overWrite>true</overWrite>
                                    <includes>org/**/*.txt</includes>
                                    <!-- apply shading: -->
                                    <outputDirectory>${commons-codec-res-folder}/${shading.prefix}</outputDirectory>
                                </artifactItem> -->
                                <!-- commons-codec resources (in META-INF) -->
                                <artifactItem>
                                    <groupId>commons-codec</groupId>
                                    <artifactId>commons-codec</artifactId>
                                    <!-- the project version specifies the commons-codec version to use: -->
                                    <version>${project.version}</version>
                                    <classifier>sources</classifier>
                                    <overWrite>true</overWrite>
                                    <includes>META-INF/*.txt</includes>
                                    <outputDirectory>${commons-codec-res-folder}</outputDirectory>
                                </artifactItem> -->
                            </artifactItems>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <!-- compile commons-codec sources -->
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${compiler-plugin-version}</version>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>${jar-plugin-version}</version>
                <executions>
                    <execution>
                        <!-- jar unshaded classes (& resources) -->
                        <id>jar-unshaded</id>
                        <phase>package</phase>
                        <goals>
                            <goal>jar</goal>
                        </goals>
                    </execution>
                    <execution>
                        <!-- rejar shaded classes (& resources), with proper manifest partially generated by bundle plugin -->
                        <id>jar-shaded</id>
                        <!-- runs after bundle plugin has done its work to generate bundle manifest -->
                        <phase>post-integration-test</phase>
                        <goals>
                            <goal>jar</goal>
                        </goals>
                        <configuration>
                            <archive>
                                <manifestFile>${manifest.path}</manifestFile>
                                <manifestEntries>
                                    <Specification-Title>${project.name}</Specification-Title>
                                    <Specification-Version>${project.version}</Specification-Version>
                                    <Specification-Vendor>${project.organization.name}</Specification-Vendor>
                                    <Implementation-Title>${project.name}</Implementation-Title>
                                    <Implementation-Version>${project.version}</Implementation-Version>
                                    <Implementation-Vendor>${project.organization.name}</Implementation-Vendor>
                                    <Implementation-Vendor-Id>org.apache</Implementation-Vendor-Id>
                                    <Implementation-Build>${implementation.build}</Implementation-Build>
                                    <X-Compile-Source-JDK>${maven.compiler.source}</X-Compile-Source-JDK>
                                    <X-Compile-Target-JDK>${maven.compiler.target}</X-Compile-Target-JDK>
                                </manifestEntries>
                            </archive>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <!-- attach sources jar -->
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-source-plugin</artifactId>
                <version>${source-plugin-version}</version>
                <configuration>
                    <archive>
                        <manifest>
                            <addDefaultImplementationEntries>true</addDefaultImplementationEntries>
                            <addDefaultSpecificationEntries>true</addDefaultSpecificationEntries>
                        </manifest>
                    </archive>
                </configuration>
                <executions>
                    <execution>
                        <!-- jar unshaded sources -->
                        <id>attach-unshaded-sources</id>
                        <!-- <phase>package</phase> (default) -->
                        <goals>
                            <goal>jar</goal>
                        </goals>
                    </execution>
                    <execution>
                        <!-- rejar shaded sources -->
                        <id>attach-shaded-sources</id>
                        <phase>post-integration-test</phase>
                        <goals>
                            <goal>jar</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <!-- apply the shading to main jar and sources jar -->
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>${shade-plugin-version}</version>
                <executions>
                    <execution>
                        <id>shading-main-jar-and-sources-jar</id>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <!-- (not needed as it is the one and only artifact/dependency)
                            <artifactSet> 
                                <includes>
                                    <include>commons-codec:*</include>
                                </includes>
                            </artifactSet>
                            -->
                            <relocations>
                                <relocation>
                                    <pattern>${commons-codec-package}</pattern>
                                    <shadedPattern>${shaded-commons-codec-package}</shadedPattern>
                                </relocation>
                            </relocations>
                            <createDependencyReducedPom>false</createDependencyReducedPom>
                            <!-- (only needed when dependency reduced pom is generated)
                            <dependencyReducedPomLocation>${project.build.directory}/dependency-reduced-pom.xml</dependencyReducedPomLocation> 
                            <keepDependenciesWithProvidedScope>true</keepDependenciesWithProvidedScope> 
                            <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
                            -->
                            <createSourcesJar>true</createSourcesJar>
                            <shadeSourcesContent>true</shadeSourcesContent>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-antrun-plugin</artifactId>
                <version>${antrun-plugin-version}</version>
                <executions>
                    <execution>
                         <!-- unpack shaded classes & sources for manifest generation and re-jarring -->
                        <id>post-shading-tasks</id>
                        <phase>package</phase>
                        <goals>
                            <goal>run</goal>
                        </goals>
                        <configuration>
                            <target>
                                <!-- Unjar shaded classes for generation of manifest -->
                                <echo>Deleting unshaded classes...</echo>
                                <delete dir="${project.build.directory}/classes"/>
                                <echo>Unjarring shaded main jar...</echo>
                                <unzip src="${project.build.directory}/${project.artifactId}.jar" dest="${project.build.directory}/classes"/>
                                <!-- delete to prevent dual inclusion in new main jar -->
                                <delete dir="${project.build.directory}/classes/META-INF/maven"/>
                                <!-- Unjar shaded sources -->
                                <echo>Deleting unshaded sources...</echo>
                                <delete dir="${commons-codec-src-folder}"/>
                                <echo>Unjarring shaded sources jar...</echo>
                                <unzip src="${project.build.directory}/${project.artifactId}-sources.jar" dest="${commons-codec-src-folder}"/>
                                <!-- delete to prevent dual inclusion in new sources jar -->
                                <delete dir="${commons-codec-src-folder}/META-INF"/>
                            </target>
                        </configuration>
                    </execution>
                    <execution>
                            <id>delete-orginals</id>
                            <phase>verify</phase>
                            <goals>
                                <goal>run</goal>
                            </goals>
                            <configuration>
                                <target>
                                    <echo>Deleting unshaded jar files...</echo>
                                    <delete>
                                        <fileset dir="${project.build.directory}" includes="**/original-*.jar" />
                                    </delete>
                                </target>
                            </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <!-- taken/modified from: http://svn.apache.org/repos/asf/commons/proper/commons-parent/trunk/pom.xml -->
                <groupId>org.apache.felix</groupId>
                <artifactId>maven-bundle-plugin</artifactId>
                <version>${bundle-plugin-version}</version>
                <configuration>
                    <archive>
                        <forced>true</forced>
                    </archive>
                    <excludeDependencies>true</excludeDependencies>
                    <manifestLocation>${project.build.directory}</manifestLocation>
                    <instructions>
                        <!-- stops the "uses" clauses being added to "Export-Package" manifest entry -->
                        <_nouses>true</_nouses>
                        <!-- Stop the JAVA_1_n_HOME variables from being treated as headers by Bnd -->
                        <_removeheaders>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME</_removeheaders>
                        <Bundle-SymbolicName>${commons.osgi.symbolicName}</Bundle-SymbolicName>
                        <Export-Package>${commons.osgi.export}</Export-Package>
                        <Private-Package>${commons.osgi.private}</Private-Package>
                        <Import-Package>${commons.osgi.import}</Import-Package>
                        <DynamicImport-Package>${commons.osgi.dynamicImport}</DynamicImport-Package>
                        <Bundle-DocURL>${project.url}</Bundle-DocURL>
                    </instructions>
                </configuration>
                <executions>
                    <execution>
                        <id>bundle-manifest</id>
                        <!-- runs after the unjarring of the shaded classes -->
                        <phase>integration-test</phase><!--  default is: process-classes -->
                        <goals>
                            <goal>manifest</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

And in the (apk/apklib/aar/jar) project(s) where you want to use the shaded library:

<!-- ... -->
<dependency>
    <groupId>commons-codec</groupId>
    <artifactId>commons-codec-shaded</artifactId>
    <version>1.9</version>
</dependency>
<!-- ... -->

Solution 3

This is due to a namespace collision resulting from and old (1.2) version of Commons Codec that comes packaged with Android colliding with your newer version. While shading is a good workaround, I don't think it is a sustainable solution in the long run. This is a systemic problem that can arise with any open source library that is packaged in Android. I've submitted an issue to Google. If you agree, I encourage you to "star" it so that it gets the attention it needs. Here's the link - https://code.google.com/p/android/issues/detail?id=160578

11,958

nbe_42

Updated on October 10, 2022

Comments

nbe_42 over 1 year
I have a problem with Android libraries.

I would like use the method Hex.encodeHexString(Byte Array) from the library org.apache.commons.codec.binary.Hex (version 1.6)

On my Android platform (SDK 2.3.1), the commons-codec library version 1.3 already exist but the method doesn't exist yet in this version (only encodeHex() ).

I added the jar library of version 1.6 into my Eclipse project (into /libs directory) but when I run the project on Emulator, I get this :
```
E/AndroidRuntime(1632): FATAL EXCEPTION: main
E/AndroidRuntime(1632): java.lang.NoSuchMethodError: org.apache.commons.codec.binary.Hex.encodeHexString
```
How can I indicate the OS where is the good library?

I'm using Eclipse Juno with Java 1.6.0 on Mac OS X

Sorry for my bad english and thanks in advance!

EDIT : My problem could be apparently solved with jarjar tool. http://code.google.com/p/google-http-java-client/issues/detail?id=75

Someone could help me with this tool? I don't know how to create an Ant Manifest or a jar file.

Thanks
- Dror Fichman almost 11 years
  
  As you mentioned - a workaround for this issue is to use jarjar to create a new jar which will not conflict with android's classes. An explanation + solution + created jar (that solved the problem for me) - priyanka-tyagi.blogspot.co.il/2013/03/… (thanks to bianca who saved us time...)
nbe_42 over 11 years

The library is added to the build path. The building work fine but my app crash on the runtime. I've already trying to use the SDK of JB or ICS but same problems.. Thanks
Aiden Fry about 10 years

This looks like a great answer. Yet, when I add this into the Maven build script the resulting APK wont install. It is saying [INSTALL_PARSE_FAILED_NO_CERTIFICATES] and there i use jarsigner to verify the build it says "jarsigner: java.lang.SecurityException: Invalid signature file digest for Manifest main attributes" Any ideas why?
nbe_42 about 10 years

Your maven build config doesn't sign the APK. I use the maven-jarsigner-plugin for doing this: <plugin> <artifactId>maven-jarsigner-plugin</artifactId> <version>1.2</version> <inherited>true</inherited> </plugin> You should also be able to sign it with the debug key by configuring correctly the maven-android-plugin.
Aiden Fry about 10 years

I have the jarsigner configured, this build script has been working for months now. I have included the Shade plugin below my maven-jarsigner-plugin in the build executions, and now the output SHADED apk is unsigned.
nbe_42 about 10 years

Okay, Did you configure you jarsigner plugin correctly ? By adding the right include ? <includes> <include>${project.build.directory}/${pro‌ject.artifactId}.apk‌</include><include>$‌{project.build.direc‌tory}/${project.arti‌factId}-SHADED.apk</‌include> </in‌cludes>
Aiden Fry about 10 years

Thanks for the pointer - I have tried this, and I just tried signing the file manually using the jarsigner tool in terminal, still hitting the same problem when installing. Looking in logcat it seems to be failing because it can not find any certificates for my assets/mydatabase.sqlite files. Not sure it this is relevant or just the first thing it checks. I will come back when I figure it out. thanks for the help.
Aiden Fry about 10 years

In the end I gave up chasing around Maven plugins and I followed this stackoverflow.com/a/21672644/1312937 to manually recreate the Jar. I then had to place this new jar file into my .m2/repositary and update the maven dependancy to point at this new file. Done in 5 minutes. Thanks again nbe, yours is the proper solution if i could get it to work I would have used it. But i don't have days to spend on build scripts ;-) A
Atorian over 9 years

Really awesome out-of-the-box solution. I was "shading" manually before, which worked fine, but this is so much more flexible. The convo continues here: github.com/simpligility/android-maven-plugin/issues/487 Google should deal with this issue (say a compiler patch?), or at least provide a solution like this.
nbe_42 about 9 years

Unfortunately Commons-codec is not the only one legacy library that is used in Android. There are a lot more (XML, JSON, Apache HTTP Client, Javax, bouncycastle, ...). In my case, the shading is the best solution since I've modularized my software in order to have common code between my server and android. I can have the same code running with the same behavior in both platforms.