How can I host my own private conda repository?

Question

I have a few python projects that are dependent on each other. I have different release versions for each project and different projects might be dependent on different release versions of a particular project. I would like to create my own conda repository on an internal server where I can push the releases of these projects as conda packages and the other projects can install the required version from there. Is this possible? If so how?

Solution 1

You can use a conda custom channel as your private repo. The essential steps are to use "conda build" to create a conda package, then copy that package into your custom channel (a directory), and now run conda index on that directory. You can then install packages from this channel by using the "conda install -c ".

An example, in more detail, let's assume linux-64:

• Create the channel:
  mkdir -p /tmp/my-conda-channel/linux-64

• Now assuming you have some project named "abc" with a meta.yaml and build.sh with some version X. Now you build it:

  conda build abc

• This will build a tar.bz2 file in your conda-bld directory. For example: ~/miniconda3/conda-bld/linux-64/abc-X-py35_0.tar.bz2. Copy that file to your channel:

  cp ~/miniconda3/conda-bld/linux-64/abc-X-py35_0.tar.bz2 /tmp/my-conda-channel/linux-64/

• Now index it:

  conda index /tmp/my-conda-channel/linux-64/

You've now uploaded that package to your custom channel. You can install it in any of your conda environments by doing:

conda install -c file://tmp/my-conda-channel/ abc=X

Where, recall, the X is the version so, once you've placed more versions in your channel, you can install specific versions.

If you have a project that depends on the X version of "abc" then we simply add it to that projects meta.yaml. Example:

package:
  name: some-other-project
  version: 0.1
requirements:
  build:
   - abc X
...

Once you have created this channel it's probably a good idea to add it to your .condarc file so that it will get automatically searched. For example:

channels:
- file://tmp/my-conda-channel/   
- defaults

Solution 2

There are two parts to this: how to create the channel and how to use it. Part two is the hardest to do nicely.

The first part is described in detail in the conda documentation. You can serve the channel directly from files, or via a static webserver.

To use the channel, one approach is the -c file://tmp/my-conda-channel/, but recent conda versions allow a much better solution via custom channels which where (recently?) added to conda.

The documentation is available via conda config --describe which includes this part:

# custom_channels (map: str)
#   A map of key-value pairs where the key is a channel name and the value
#   is a channel location. Channels defined here override the default
#   'channel_alias' value. The channel name (key) is not included in the
#   channel location (value).  For example, to override the location of
#   the 'conda-forge' channel where the url to repodata is
#   https://anaconda-repo.dev/packages/conda-forge/linux-64/repodata.json,
#   add an entry 'conda-forge: https://anaconda-repo.dev/packages'.
#
# custom_channels: {}

The syntax for adding a channel is not documented, but reading the source the correct invocation is seen to be:

conda config --set custom_channels.my-conda-channel file://tmp/

(Note: my-conda-channel/ is not part of path). With this added to your config, you can now use your own channel in the same way you would conda-forge or other 'built-in' channels:

conda install -c my-conda-channel my-cool-package

For anyone in an MS Windows setting, the correct set of slashes and backslashes for using this with a windows share is file://\\SOMECORP\Corp\conda\channels\. Works a charm.

Solution 3

If you want to add the channel on Windows, try:

conda config --append channels file:///C:\tmp\my-conda-channel

Make sure, you followed the instructions from the Paul's and Janus' answer.

Comments

[Leo Gallucci]

Did you have a solution for https hosting? or even S3?

[Joey Baruch]

I just got a free tier from jfrog

[ostrokach]

Any way to do this so that your channel is accessible over http?

[Machiel]

This is useful information but doesn't answer the question completely, the question is how to get an (http) server up and running so that other machines can pull the packages from there.

[Roland Weber]

@Machiel The question mentions an internal server, but not http. The answer can be used with an internal file server.

[Roland Weber]

Here's a blog about running nginx in a Docker container to serve conda channels. cityscience.com/blog/private-conda-channel.html

[Jeff Tilton]

Can this be extended to include python interpreters as well? I need to have a completely private environment management.

[kalebo]

@RolandWeber 's link has rotted: here's the updated link cityscience.com/news/2017-02-09-private-conda-channel