how do I get sbt to use a local maven proxy repository (Nexus)?

Question

I've got an sbt (Scala) project that currently pulls artifacts from the web. We'd like to move towards a corporate-standardized Nexus repository that would cache artifacts. From the Nexus documentation, I understand how to do that for Maven projects. But sbt obviously uses a different approach. (I understand Ivy is involved somehow, but I've never used it and don't understand how it works.)

How do I tell sbt and/or the underlying Ivy to use the corporate Nexus repository system for all dependencies? I'd like the answer to use some sort of project-level configuration file, so that new clones of our source repository will automatically use the proxy. (I.e., mucking about with per-user config files in a dot-directory is not viable.)

Thanks!

Solution 1

Step 1: Follow the instructions at Detailed Topics: Proxy Repositories, which I have summarised and added to below:

1. (If you are using Artifactory, you can skip this step.) Create an entirely separate Maven proxy repository (or group) on your corporate Maven repository, to proxy ivy-style repositories such as these two important ones:

   • http://repo.typesafe.com/typesafe/ivy-releases/
   • http://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/

   This is needed because some repository managers cannot handle Ivy-style and Maven-style repositories being mixed together.

2. Create a file repositories, listing both your main corporate repository and any extra one that you created in step 1, in the format shown below:

   [repositories]
     my-maven-proxy-releases: http://repo.example.com/maven-releases/
     my-ivy-proxy-releases: http://repo.example.com/ivy-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
   

3. Either save that file in the .sbt directory inside your home directory, or specify it on the sbt command line:

   sbt -Dsbt.repository.config=<path-to-your-repo-file>
   

Good news for those using older versions of sbt: Even though, in the sbt 0.12.0 launcher jar at least, the boot properties files for older sbt versions don't contain the required line (the one that mentions repository.config), it will still work for those versions of sbt if you edit those files to add the required line, and repackage them into the sbt 0.12.0 launcher jar! This is because the feature is implemented in the launcher, not in sbt itself. And the sbt 0.12.0 launcher is claimed to be able to launch all versions of sbt, right back to 0.7!

Step 2: To make sure external repositories are not being used, remove the default repositories from your resolvers. This can be done in one of two ways:

1. Add the command line option -Dsbt.override.build.repos=true mentioned on the Detailed Topics page above. This will cause the repositories you specified in the file to override any repositories specified in any of your sbt files. This might only work in sbt 0.12 and above, though - I haven't tried it yet.
2. Use fullResolvers := Seq( resolver(s) for your corporate maven repositories ) in your build files, instead of resolvers ++= or resolvers := or whatever you used to use.

Solution 2

OK, with some help from Mark Harrah on the sbt mailing list, I have an answer that works.

My build class now looks like the following (plus some other repos):

import sbt._

//By extending DefaultWebProject, we get Jetty support
class OurApplication(info: ProjectInfo) extends DefaultWebProject(info) {

  // This skips adding the default repositories and only uses the ones you added
  // explicitly. --Mark Harrah
  override def repositories = Set("OurNexus" at "http://our.nexus.server:9001/nexus/content/groups/public/") 
  override def ivyRepositories = Seq(Resolver.defaultLocal(None)) ++ repositories

  /* Squeryl */
  val squeryl = "org.squeryl" % "squeryl_2.8.0.RC3" % "0.9.4beta5"

  /* DATE4J */
  val date4j = "hirondelle.date4j" % "date4j" % "1.0" from "http://www.date4j.net/date4j.jar"

  // etc
}

Now, if I delete the Squeryl tree from my machine's .ivy2/cache directory, sbt tries to grab it from the Nexus tree with the appropriate URL. Problem solved!

Solution 3

All you need is to define a property file sbt.boot.properties which will allow you to:

• redefine the ivy cache location (I need that because it would be otherwise part of our roaming Windows profile, which is severely limited in disk space in our shop. See Issue 74)
• define any other Maven repo you want

    C:\HOMEWARE\apps\sbt-0.74\sbt.boot.properties

    [scala]
      version: 2.7.7
    #  classifiers: sources, javadoc

    [app]
      org: org.scala-tools.sbt
      name: sbt
      version: read(sbt.version)
      class: sbt.xMain
      components: xsbti
      cross-versioned: true
      classifiers: sources, javadoc

    [repositories]
      local
      my-nexus: http://my.nexus/nexus/content/repositories/scala-tools/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
      maven-local
    #  sbt-db: http://databinder.net/repo/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
    #  maven-central
    #  scala-tools-releases
    #  scala-tools-snapshots

    [boot]
     directory: project/boot
     properties: project/build.properties
     prompt-create: Project does not exist, create new project?
     prompt-fill: true
     quick-option: true

    [log]
     level: debug

    [app-properties]
     project.name: quick=set(test), new=prompt(Name)[p], fill=prompt(Name)
     project.organization: new=prompt(Organization)[org.vonc]
     project.version: quick=set(1.0), new=prompt(Version)[1.0], fill=prompt(Version)[1.0]
     build.scala.versions: quick=set(2.8.0.RC2), new=prompt(Scala version)[2.8.0.RC2], fill=prompt(Scala version)[2.8.0.RC2]
     sbt.version: quick=set(0.7.4), new=prompt(sbt version)[0.7.4], fill=prompt(sbt version)[0.7.4]
     project.scratch: quick=set(true)
     project.initialize: quick=set(true), new=set(true)

    [ivy]
     cache-directory: C:\HOMEWARE\projects\.ivy2\cache

Note: this sbt.boot.properties file is inspired from:

• the one mentioned in the "Generalized Launcher" page of the sbt project.
• the one found within sbt-0.74 itself!

I have commented any external Maven repository definition, and added a reference to my own Nexus Maven repo.

The launcher may be configured in one of the following ways in increasing order of precedence:

• Replace the /sbt/sbt.boot.properties file in the jar.
• Put a configuration file named sbt.boot.properties on the classpath. Put it in the classpath root without the /sbt prefix.
• Specify the location of an alternate configuration on the command line. This can be done by:
  • either specifying the location as the system property sbt.boot.properties
  • or as the first argument to the launcher prefixed by '@'.

The system property has lower precedence.
Resolution of a relative path is:

• first attempted against the current working directory,
• then against the user's home directory,
• and then against the directory containing the launcher jar.

An error is generated if none of these attempts succeed.

---

Define a sbt.bat wrapper (in order to be sure to specify your sbt.boot.properties) like:

C:\HOMEWARE>more C:\HOMEWARE\bin\sbt.BAT
@echo off
set t=%~dp0
set adp0=%t:C:\="%"

set SBT_DIR=%adp0%..\apps\sbt-0.74
dir C:\%SBT_DIR%\sbt-launch-0.7.4.jar
# if needed, add your proxy settings
set PROXY_OPTIONS=-Dhttp.proxyHost=my.proxy -Dhttp.proxyPort=80xx -Dhttp.proxyUser=auser -Dhttp.proxyPassword=yyyy
set JAVA_OPTIONS=-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m -Xmx512M -cp C:\HOMEWARE\apps\sbt-0.74\sbt-launch-0.7.4
set SBT_BOOT_PROPERTIES=-Dsbt.boot.properties="sbt.boot.properties"
cmd /C C:\HOMEWARE\apps\jdk4eclipse\bin\java.exe %PROXY_OPTIONS% %JAVA_OPTIONS% %SBT_BOOT_PROPERTIES% -jar C:\HOMEWARE\apps\sbt-0.74\sbt-launch-0.7.4.jar %*

And your sbt will download artifacts only from:

• your Nexus
• your local Maven repo.

---

Just tested at home with an old Nexus opensource 1.6 I had running, java 1.6, sbt07.4

C:\Prog\Java\jdk1.6.0_18\jre\bin\java  -Xmx512M -Dsbt.boot.properties=sbt.boot.properties - jar "c:\Prog\Scala\sbt\sbt-launch-0.7.4.jar"  

That gives:

[success] Build completed successfully.
C:\Prog\Scala\tests\pp>sbt
Getting Scala 2.8.0 ...
downloading http://localhost:8081/nexus/content/repositories/scala/org/scala-lang/scala-compiler/2.8.0/scala-compiler-2.
8.0.jar ...
        [SUCCESSFUL ] org.scala-lang#scala-compiler;2.8.0!scala-compiler.jar (311ms)
downloading http://localhost:8081/nexus/content/repositories/scala/org/scala-lang/scala-library/2.8.0/scala-library-2.8.
0.jar ...
        [SUCCESSFUL ] org.scala-lang#scala-library;2.8.0!scala-library.jar (185ms)
:: retrieving :: org.scala-tools.sbt#boot-scala
        confs: [default]
        2 artifacts copied, 0 already retrieved (14484kB/167ms)
[info] Building project test 0.1 against Scala 2.8.0
[info]    using sbt.DefaultProject with sbt 0.7.4 and Scala 2.7.7

If I try a funny value in the sbt.boot.properties file:

C:\Prog\Scala\tests\pp>sbt
Getting Scala 2.9.7 ...

:: problems summary ::
:::: WARNINGS
                module not found: org.scala-lang#scala-compiler;2.9.7
        ==== nexus: tried
          http://localhost:8081/nexus/content/repositories/scala/org/scala-lang/scala-compiler/2.9.7/scala-compiler-2.9.7.pom
          -- artifact org.scala-lang#scala-compiler;2.9.7!scala-compiler.jar:
          http://localhost:8081/nexus/content/repositories/scala/org/scala-lang/scala-compiler/2.9.7/scala-compiler-2.9.7.jar

So it does limit itself to the two repo I defined:

[repositories]
nexus:  http://localhost:8081/nexus/content/repositories/scala
nexus2: http://localhost:8081/nexus/content/repositories/scala, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]

(I commented everything else: local, maven-local, ...)

If I comment all repositories and put a funny value (2.7.9) for the scala version in the sbt.boot.properties, I do get (like the OP did)

C:\Prog\Scala\tests\pp>sbt
Error during sbt execution: No repositories defined.

If I put 2.7.7 (while still having all repo commented), yes, it won't generate an error:

C:\Prog\Scala\tests\pp>sbt
[info] Building project test 0.1 against Scala 2.8.0
[info]    using sbt.DefaultProject with sbt 0.7.4 and Scala 2.7.7

But that's only because it already had downloaded scala2.8.0 during my previous tries.
If I remove that library from my project/boot directory, then it will throw an Exception:

[info]    using sbt.DefaultProject with sbt 0.7.4 and Scala 2.7.7
> C:\Prog\Scala\tests\pp>sbt
Error during sbt execution: No repositories defined.
        at xsbt.boot.Pre$.error(Pre.scala:18)
        at xsbt.boot.Update.addResolvers(Update.scala:197)
...
        at xsbt.boot.Boot$.main(Boot.scala:15)
        at xsbt.boot.Boot.main(Boot.scala)
Error loading project: Error during sbt execution: No repositories defined.

Solution 4

edit the config file in sbt_home/conf "sbtconfig.txt"

add two line

-Dsbt.override.build.repos=true
-Dsbt.repository.config="C:/Program Files (x86)/sbt/conf/repo.properties"

the repo.properties content is

[repositories]
    local
    public: http://222.vvfox.com/public  <-fix this ,write your local nexus group url

Solution 5

Well this has bugged me for a while so I found a guy that has written an SBT plugin for maven out on github called maven-sbt so all you have to do is include it in your plugins project and make your project mixin with maven.MavenDependencies and all your operations like update and publish-local work with your local maven. The nice thing about that is if you are like me, your org is all maven. So, all you libs are in you local maven repo but if for some reason you build with sbt first, then you start getting a bunch or jars in ivy too. What a waste of space, and time since you will still need to get them for your maven builds.

That said, I wish this were built into sbt so I would not need to add it to every project. Maybe as a processor at least. He mentioned in one thing I read that he would like to add it to 0.9 but I have not been able to find it.

View more solutions

Comments

[Devanshu Mevada]

@VonC I did some bulk update of questions with the [maven] tag and thought this one was more about Nexus than Maven(-2). But if you think it's relevant, feel free to rollback/update accordingly, I went maybe too fast on this one.

[Harlan]

yeah, maven is maybe not the primary tag, but it's certainly about maven repos. I rolled it back.

[VonC]

By the way, it is confirmed: it works as advertised, both at home and at work. I have edited my question to illustrate that, and to add some repository definitions for you to examine.

[Harlan]

Thanks for the extremely complete response! But it's not working for me. I've created an sbt.boot.properties file, and it only has three repositories listed: local, maven-local, and my-nexus which points to our local Nexus repo. sbt is a script with the following: java -Xmx1024M -Dsbt.boot.properties="sbt.boot.properties" -jar dirname $0/sbt-launch.jar "$@" If I then delete a package (say, Squeryl) from my .ivy2 cache, then do "sbt update", it appears to be pulling from the public internet, without touching our Nexus install. How/why?

[VonC]

@Harlan: Why? because it doesn't take into your account your sbt.boot.properties. Did you create it just beside sbt-launch-0.7.4.jar (meaning in your classpath)? Come to think of it, I don't see any classpath explicitly defined in your java command.

[Harlan]

It's definitely loading sbt.boot.properties. With the command I showed above, if I change the Scala version to 2.9.7 in that file, it vainly tries to get 2.9.7. So it's not a classpath problem. But sbt update is pulling from scala-tools.org even if only local and my-nexus are defined.

[VonC]

@Harlan: did you try to define no repositories? Or to define something obviously false? I still have a hard time believing your specific property file is taken into account.

[Harlan]

This is weird. If I set scala version to 2.9.7, and comment out all repositories, then it gripes "Error during sbt execution: No repositories defined." But if I set scala version to 2.7.7, and comment out all repositories, it gripes and seemingly ignores the repository list!

[Harlan]

To clarify, does the sbt.boot.properties file affect the repositories that sbt uses to build your project, or just the repositories it uses to pull Scala and other artifacts so it itself can run?

[VonC]

@Harlan: the "[scala] version: 2.7.7" should definitively stay at 2.7.7 and has nothing to do with the Scala version your program will actually be compiled with. The other Scala versions can be tweaked as default value when you create the project. I am not sure why "2.9.7" even comes in play, since there is no Scala 2.9.7...

[Harlan]

I just did that to prove that the file was being read. Now that I know it is, I see no evidence that your solution actually affects the way sbt pulls artifacts for my project.

[VonC]

@Harlan: Sorry you didn't make it work. I will retest it tomorrow at work. Just wondering, do you have some Maven settings.xml that could be taken into account in this process and could also point directly to some internet repos?

[Harlan]

Thanks, VonC. No, there are no settings.xml files anywhere that could be interfering with this process.

[VonC]

@Harlan: I just made a quick test at home before bedtime. It works as advertised. I will confirm tomorrow with more recent settings (jdk1.6_21, NexusPro1.7.2 soon to be 1.8). Look at the definition of my repositories. Maybe there is something different with your definition? In any case, you can complete your question with the exact sbt.boot.properties file you have if you want (for me to test on my side).

[Harlan]

Thanks, Von. I got an official answer from the sbt mailing list, which I'll put on a different thread. Turns out the solution you provided does in fact only apply to the bootstrapping process, not to the project itself.

[VonC]

@Harlan: very interesting. Could you post it as a separate answer in this thread?

[Harlan]

sorry, that's what I meant. look below.

[VonC]

@Harlan. Looked and +1. Still prefer my way: no modification within the project: every member of my team will use the sbt.boot.properties I provide and won't have to make any extra modification to use our corporate Nexus.

[VonC]

@Harlan: doesn't work for you. But I suspect those two repository definitions are not for the same kind of update, so the two solution might be complementary. I thank you for the opportunity to document mine.

[Robin Green]

I can confirm that this doesn't work. At all. I've filed a bug report.

[VonC]

@RobinGreen: interesting. However I don't see in your bug report the exact version of sbt, scala and java you are using. (and my answer was for an sbt more than 18 months older)

[delitescere]

This is now out of date for Play 2 Final and SBT 0.11.2.

[Totoro]

I'm using Scala 2.10, Artifactory 3, sbt 12.3 and having no luck using the directions here and in the docs referenced herein. I know its reading repositories after finding a typeo there. I've set the property on the launcher via -D. SBT is acting like its not disregarding the default resolvers as it says it should with -Dsbt.override.build.repos=true. I could try point 2 above, but it sounds like it should not be neccessary?

[Grundlefleck]

@Robin Green: thanks so much, this problem was driving me crazy and your solution worked for me.

[F30]

If you want to configure this per project (as asked in the original question) and don't want to add the command line options to every sbt run, you may create a file called ".sbtopts" next to your "build.sbt". Put the options (-Dsbt.override.build.repos=true -Dsbt.repository.config=<…>) in it and they will get picked up by the sbt executable.