how to publish 3rdparty artifacts with ivy and nexus

Question

I'm busily getting my feet wet with ivy. I have an existing nexus repository running on my local PC, and an existing ant build script.
Both work fine.

Part of the build scripts have some files to retrieve our 3rdparty jar files (log4j, xmlbeans, junit, pdf, etc..) from a network share - which is klunky at best.

I want to use ivy's dependency mechanisms to retrieve these files from a nexus repository and use that in the build. Each 3rdparty lib has a name and an arbitrary set of files (jar, dll, license.dat, xml, etc).

Since we have a large number of these 3rdparty libs and each lib has multiple files - manual uploading to nexus is not an option -- i need something i can use to take a set of files, give them a lib name, a version number and upload the result to nexus. then I need to be able to retrieve this from ivy.

I managed to get the upload part to work, but the retreival process does not work. Using our xmlbeans lib as a starting point I created the following ivy.xml file

<ivy-module version="1.0">  
<info organisation="thirdparty_tools" module="xmlbeans" status="integration">  
<publications>  
  <artifact name="jsr173_api" type="jar" ext="jar"/>  
  <artifact name="saxon-dom" type="jar" ext="jar"/>  
  <artifact name="saxon-xpath" type="jar" ext="jar"/>  
  <artifact name="saxon" type="jar" ext="jar"/>  
  <artifact name="xbean" type="jar" ext="jar"/>  
  <artifact name="xbean_xpath" type="jar" ext="jar"/>  
  <artifact name="xmlpublic" type="jar" ext="jar"/>  
</publications>  
</ivy-module>  

and then some ant script to publish it to nexus:

 <ivy:resolve/>  
    <ivy:publish <ivy:publish resolver="thirdparty" forcedeliver="true" update="true" revision="${version}" overwrite="true">  
      <artifacts pattern="[artifact].[ext]"/>  
    <ivy:publish/>  

And this all works fine. It publishes all the jar files to nexus in the expected directory.

The trouble comes when I try to use it in my build. I created the following ivy.xml file for my build:

<ivy-module version="1.0">  
    <info organisation="myCompany" module="GLB_Data"/>  
    <dependencies>  
        <dependency org="thirdparty_tools" name="xmlbeans" rev="2.2.0"/>  
    </dependencies>  
</ivy-module> 

Then when I run my build - it fails to find anything:

::::::::::::::::::::::::::::::::::::::::::::::  
::          UNRESOLVED DEPENDENCIES         ::  
::::::::::::::::::::::::::::::::::::::::::::::  
:: thirdparty_tools#jsr173_api;2.2.0: not found  
:: thirdparty_tools#saxon-dom;2.2.0: not found  
:: thirdparty_tools#saxon-xpath;2.2.0: not found  
:: thirdparty_tools#saxon;2.2.0: not found  
:: thirdparty_tools#xbean;2.2.0: not found  
:: thirdparty_tools#xbean_xpath;2.2.0: not found  
:: thirdparty_tools#xmlpublic;2.2.0: not found  
:::::::::::::::::::::::::::::::::::::::::::::: 

The problem seems to be with this pattern:

WARN: ==== public: tried  
WARN:   http //localhost:8081/nexus/content/groups/public/thirdparty_tools/jsr173_api/2.2.0/jsr173_api-2.2.0.pom  
WARN:   -- artifact thirdparty_tools#jsr173_api;2.2.0!jsr173_api.jar:  
WARN:   http //localhost:8081/nexus/content/groups/public/thirdparty_tools/jsr173_api/2.2.0/jsr173_api-2.2.0.jar  

ivy seems to be looking for the jsr173_api artifact under its own name, rather than under the xmlbeans folder where it was published to:  
[ivy:publish]   published jsr173_api to http //localhost:8081/nexus/content/repositories/thirdparty/thirdparty_tools/xmlbeans/2.2.0/jsr173_api-2.2.0.jar  

(urls obfuscated to prevent accidents).

so somehow I need to publish differently, or retrieve differently. Ideas and suggestions are much appreciated.

Solution

Nexus is primarily a Maven repository, this means one must adapt to the way Maven structures artifacts.

Since you're focused on bulk loading Nexus I suggest looking at the answer to the following question:

Upload artifacts to Nexus, without Maven

If you wish to stick with ivy read on.....

Background

Need a Maven POM

Your first issue is that your Maven module(s) will need a POM file. This file describes the maven module and can be easily generated from the contents of your ivy.xml file (See solution below).

Secondly, Maven assumes that there is one primary artifact being built. For example:

<project>
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.myspotontheweb</groupId>
  <artifactId>donaldduck</artifactId>
  <version>1.0.1</version>
  <packaging>txt</packaging>
</project>

A Maven client would translate this information into the following URL:

http://<host>/<repo>/com/myspotontheweb/donaldduck/1.0.1/donaldduck-1.0.1.txt

This demonstrates how Nexus can store any type of binary dependency. The packaging parameter defaults to "jar".

How maven handles additional module artifacts

While Maven focuses on a single build artifact, it is possible to add additional supplementary artifacts by posting them into the same directory (as you've done).

These are not listed in the Maven POM. Instead a Maven uses a special "classifier" attribute. The following is a possible dependency declaration.

<dependency>
  <groupId>com.myspotontheweb</groupId>
  <artifactId>donaldduck</artifactId>
  <version>1.0.1</version>
  <classifier>metadata</classifier>
  <type>n3</type>
</dependency>

A Maven client would translate this into the following URL:

http://<host>/<repo>/com/myspotontheweb/donaldduck/1.0.1/donaldduck-1.0.1-metadata.n3

Open source projects typically release their source code in this manner.

Ivy Solution

So finally how does one publish files into Nexus using ivy?

First of all decide which artifact is the "main" build artifact and add an additional entry for your POM file:

<ivy-module version='2.0' xmlns:e="http://ant.apache.org/ivy/extra">

    <info organisation="com.myspotonontheweb" module="donaldduck" revision="1.0.1"/>

    <publications>
        <artifact name="donaldduck" type="txt"/>
        <artifact name="donaldduck" type="pom"/>
        <artifact name="donaldduck" type="n3" e:classifier="metadata"/>
        <artifact name="donaldduck" type="zip" e:classifier="disto"/>
    </publications>

</ivy-module>

The other files can also be listed but each must have a unique classifier attribute..... Here you will be faced with one of the classic problems translating an ANT project into Maven.... Each jar file you publish, will probably need to have a separate POM. They not really "supplementary" artifacts.....

Pretending that you don't need to publish multiple modules.... Use the following build targets to publish your module:

<target name="prepare" description="Generate POM">
    <!-- Optional: Intermediate file containing resolved version numbers -->
    <ivy:deliver deliverpattern="${build.dir}/ivy.xml" pubrevision="${publish.revision}" status="release"/>

    <!-- Generate the Maven POM -->
    <ivy:makepom ivyfile="${build.dir}/ivy.xml" pomfile="${build.dir}/donaldduck.pom"/>
</target>

<target name="publish" depends="init,prepare" description="Upload to Nexus">
    <ivy:publish resolver="nexus-deploy" pubrevision="${publish.revision}" overwrite="true" publishivy="false" >
        <artifacts pattern="${build.dir}/[artifact](-[classifier]).[ext]"/>
    </ivy:publish>
</target>

Nexus credentials

And for completeness here's the ivysettings.xml file containing the Nexus repository location and credentials:

<ivysettings>
    <settings defaultResolver="nexus-central"/>
    <credentials host="somehost" realm="Sonatype Nexus Repository Manager" username="????" passwd="????"/>
    <resolvers>
        <ibiblio name="nexus-central" root="http://somehost/nexus/content/repositories/central/" m2compatible="true"/>
        <ibiblio name="nexus-deploy" root="http://somehost/nexus/content/repositories/repo" m2compatible="true"/>
    </resolvers>
</ivysettings>

Update

Downloading artifacts

To retrieve all the published artifacts (not just the main one), you need to list them as follows:

<dependency org="com.myspotontheweb" name="donaldduck" rev="1.0.1">
    <artifact name="donaldduck" type="txt"/>
    <artifact name="donaldduck" type="n3" e:classifier="metadata"/>
    <artifact name="donaldduck" type="zip" e:classifier="distro"/>
</dependency>

Functionally the same as the following Maven fragment:

<dependency>
  <groupId>com.myspotontheweb</groupId>
  <artifactId>donaldduck</artifactId>
  <version>1.0.1</version>
  <type>txt</type>
</dependency>

<dependency>
  <groupId>com.myspotontheweb</groupId>
  <artifactId>donaldduck</artifactId>
  <version>1.0.1</version>
  <classifier>metadata</classifier>
  <type>n3</type>
</dependency>

<dependency>
  <groupId>com.myspotontheweb</groupId>
  <artifactId>donaldduck</artifactId>
  <version>1.0.1</version>
  <classifier>distro</classifier>
  <type>zip</type>
</dependency>

Comments

[Jon]

Well I forgot to say that I am somewhat familiar with how maven does things. I was hoping to find a way around the 1 artifact per module. but that aside -- I can do what you describe for single artifacts, but the main key here is that I want to somehow publish multiple files and then reference them with a single entry in my ivy.xml file in the ant build. Unless I missed something fundamental, what you gave would require me to list each file individually - which is what im trying desperately to avoid.

[Jon]

So i went and added an entry: <artifact name="xmlbeans" type="pom"/> to my ivy.xml file and re-published. It created a pom file as expected (thirdparty_tools/xmlbeans/2.2.0/xmlbeans-2.2.0.pom). However.... ivy is not looking for that. it wants a file named: thirdparty_tools/xmlbeans/2.2.0/xmlbeans-2.2.0.xml which doesnt exist. So I changed the entry to: <artifact name="xmlbeans" type="pom" ext="xml"/> and it worked as expected.

[Jon]

ok - this is not working. The errors went away on resolve, but nothing is getting downloaded when I do a retrieve. Actually only the .pom file is getting downloaded, but I want the artifacts (.zip .dll, etc).

[Mark O'Connor]

I re-tested this and it's working for me. Are you declaring the resolver correctly when downloading from the Nexus repo?

[Jon]

for the resolver I have this: <ibiblio name="thirdparty" m2compatible="true" root="http://localhost:8081/nexus/content/repositories/third‌​party/"/> As for downloading stuff - The issue seems to be that the ivy.xml file is ignored in the download process, and all ivy looks at is the pom file, and whatever dependencies it has. What really puzzles me is that ivy created the URLs to publish with.. yet ivy cannot use the same ones to download stuff. This tells me there is a fundamental flaw either in ivy's setup or in my understanding of how it works.

[Mark O'Connor]

You're using the correct resolver and you've specified that it should be Maven2 compatible. This means ivy will use the module's POM file when downloading. Isn't this what you want? Nexus is a Maven repository, I'd want to be compliant with Maven clients..... If you insist I'd suggest using a URL resolver, enabling the control you're seeking.

[Jasper Floor]

So far as I understand you cannot use an Ibiblio resolver for publishing tasks. At least according to the documentation.

[Mark O'Connor]

@JasperFloor Has worked fine for me... Where in the documentation does it say it won't work?