how to create local own pypi repository index without mirror?

Question

We have several own python packages and want to create local pypi repository for them using simple interface like https://pypi.python.org/simple/

This repository I want to create for local only without any mirrors due to security reason, and it will be put under apache's control

The command pypimirror looks has to be initialized once, which needs to mirror.

How can I generate PyPi Simple Index based on local python packages.

Any other simple scripts for this ?

Solution 1

Since you asked to answer here:

Take a look at pip2pi, it seems to be exactly what you are looking for.

Solution 2

We had a similar need at my company. Basically how can we upload "closed source" packages to an index while being able to install them as if they were on PyPI?

We have sponsored a project called devpi which acts as a PyPI cache (packages you access from PyPI will be cached on your server) as well as a powerful and fast index server. The documentation is available at http://doc.devpi.net/latest/.

Next on the roadmap is mirroring for multi geos deployment. To kick the tires on your machine takes about 5 minutes (look at the quick start guides). Finally devpi is compatible with both pip and easy_install (i.e. you do not need the devpi client installed on your machine).

Hope this help.

Solution 3

The simplest way is to organize the package distfiles into package-named dirs and run a simple HTTP server. No extra packages needed, Python's stdlib is enough. Directory structure example:

└── repodir
    ├── setuptools
    │   ├── setuptools-38.1.0-py2.py3-none-any.whl 
    │   ├── setuptools-38.1.0.zip
    │   ├── setuptools-39.2.0-py2.py3-none-any.whl 
    │   └── setuptools-39.2.0.zip
    ├── wheel
    │   └── wheel-0.31.1-py2.py3-none-any.whl 
    ...

Start the server:

$ cd repodir/
$ python3 -m http.server 9000
$ # or for Python 2:
$ python2 -m SimpleHTTPServer 9000

The local repo is up and running. Now you can pass the repo to pip:

$ pip install wheel --extra-index-url=http://127.0.0.1:9000

or even persist the repo URL in the pip.conf to not to enter it each time:

# pip.conf
[global]
extra-index-url=http://127.0.0.1:9000

Reference: Python Packaging user Guide, Hosting your own simple repository

Solution 4

There is nothing special about the mirror, and you can use mod_rewrite to set it up yourself.

1. Dump your packages in a directory that is mapped to a URL. Here I am using /url/to/my/pypi/ an an example. The folder hierarchy should be /foo/bar/simple/[name of package]/[name of tarball]

2. Add the following to .htaccess or the global configuration for that directory where you packages are. The last block of lines is a fall back to the global pypi index:

   Options +Indexes
   
   RewriteEngine On
   
   RewriteRule ^/robots.txt - [L]
   RewriteRule ^/icons/.* - [L]
   RewriteRule ^/index\..* - [L]
   
   RewriteCond /foo/bar/simple/ !-f
   RewriteCond /foo/bar/simple/ !-d
   RewriteRule ^/(.*)/?$ http://pypi.python.org/ [R,L]
   

3. Update your ~/.pip/pip.conf to point to the new repository:

   [global]
   index-url = http://localhost/url/to/my/pypi/
   

   Or use the -i http://localhost/url/to/my/pypi/ option at the command line.

Solution 5

If you are talking about running simplepypi then you will have your server for adding packages and serve them out. To quote the documentation:

- Running this on the setup.py of your favorite package:

    python setup.py sdist upload -r local

If you were to use either os.walk or glob.glob on your local site-packages directory you could quickly filter for setup.py in each of the packages/directories and invoke the above on them.

If you just need to create a directory of tar.gz files complete with a .html list of them then you can use glob.glob on the top level of your site-packages directory - tar.gz each directory in turn and add the resulting filename to a list - you can then generate your index.html from that list.

You can use any of a large number of template engines for this or generate it yourself:

import glob
filelist = glob.glob("*.tar.gz")
tags = ['<A href="file:Where/%s">%s</A>' % (s,s) for s in tags]
head = """<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<HTML>
<HEAD>
<TITLE></TITLE>
<META NAME="Generator" CONTENT="Python Script">
<META NAME="Keywords" CONTENT="Cheeseshop">
<META NAME="Description" CONTENT="List of local python packages">
</HEAD>

<BODY BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#FF0000" VLINK="#800000" ALINK="#FF00FF" BACKGROUND="?">
"""
tail = """</BODY></HTML>"""
tags.insert(0,head)
tags.append(tail)
page = "\n".join(tags)

Then save or serve you page.

View more solutions

Comments

[Piotr Dobrogost]

possible duplicate of Setting up a local PyPi server with custom set of packages

[aquavitae]

possible duplicate of How to roll my own pypi?

[Larry Cai]

this is the most simple way to handle for my case

[Larry Cai]

Little overkill for my simple case, but i vote it since it looks quite stable and easy to use.

[Bouke]

devpi is exactly what I was looking for. It acts as a pypi server, and you can use the default pip command for managing packages. Easy to setup and easy in use. For use in a virtual environment, the daemon can be started/stopped in post(de)activate hooks.

[dannysauer]

You would only do the "Options +Indexes" part of step #2 for "local only" as the question requested. But transparently redirecting to pypi for things that aren't local is also useful. :)

[Ken Williams]

It looks like pip2pi is intended for public packages that are already on PyPI, is that correct? I thought the original question was about creating a repository for packages developed in-house.

[Brian McCutchon]

This seems to be unmaintained. A major bug was reported in April, and has not been addressed by the maintainers. A PR to fix it is also going unnoticed.

[Jim]

I agree with @BrianMcCutchon. I just wasted two days trying to get pip2pi to work before discovering a bug (or bugs) that was causing it not to work for me. If you're developing a web application, I'd suggest pypiserver as that's what I ended up using. If you're developing Python applications, dev.pi would probably be a better choice. Just don't use pip2pi.

[Brian McCutchon]

@Jim Well, the PR was finally merged, and the maintainer added another maintainer, so it might be better maintained now.

[quiet_penguin]

wrote a script that generates a simple repository with N recent versions of 4000 most used packages on pypi. Advantage is it can hold multiple versions as in pypi. gist.github.com/harisankar-krishna-swamy/…