How / Is it possible to install python in a portable way?

Question

I'm working on a python script which takes care to migrate a mysql database with a certain schema / structure, into a postgresql database with a different structure.

During the development phase I was working inside a Virtual Machine(CentOS7) with all my environment correctly set up.

Currently I'm in the testing phase, and I'm trying to run the script in a real server for the first time, but I'm already facing troubles caused by different environment (python version, or python-modules version incompatibility).

Since I will have to execute this script into many servers(All of them will be GNU/Linux servers, most of them CentOS, some Debian), I'm looking for a way to integrate python, and all the python-modules (dependencies) directly into my script, a sort of portable version of python, if you know what I mean.

E.g. I would like to integrate into my script package the following elements/binaries:

• Python 2.7.5
• mysql-connector-python-2.1.3-1
• python-psycopg2

Solution

virtualenv is probably what you are looking for. See http://docs.python-guide.org/en/latest/dev/virtualenvs/:

A Virtual Environment is a tool to keep the dependencies required by different projects in separate places, by creating virtual Python environments for them. It solves the “Project X depends on version 1.x but, Project Y needs 4.x” dilemma, and keeps your global site-packages directory clean and manageable.

Install virtualenv via pip:

$ pip install virtualenv

Basic Usage

1. Create a virtual environment for a project:

   $ cd  my_project_folder 
   $ virtualenv venv 
   

   virtualenv venv will create a folder in the current directory which will contain the Python executable files, and a copy of the pip library which you can use to install other packages. The name of the virtual environment (in this case, it was venv) can be anything; omitting the name will place the files in the current directory instead.

   This creates a copy of Python in whichever directory you ran the command in, placing it in a folder named venv.

   You can also use a Python interpreter of your choice.

   $ virtualenv -p /usr/bin/python2.7 venv 
   

   This will use the Python interpreter in /usr/bin/python2.7

2. To begin using the virtual environment, it needs to be activated:

   $ source venv/bin/activate 
   

   The name of the current virtual environment will now appear on the left of the prompt (e.g. (venv)Your-Computer:your_project UserName$) to let you know that it’s active. From now on, any package that you install using pip will be placed in the venv folder, isolated from the global Python installation.

   Install packages as usual, for example:

   $ pip install requests 
   

   If you are done working in the virtual environment for the moment, you can deactivate it:

   $ deactivate
   

If you want to move your environment:

You can make a list of installed packages inside the virtualenv:

    $ pip freeze > requirements.txt

And install them on the destination virtualenv using:

    $ pip install -r requirements.txt

From my experience, virtualenvs can be created and managed for both python2 and python3 (on my system, I have both virtualenv and virtualenv3)

Note that virtualenv does not itself provide the python interpreter. It allows you to create isolated environments where a python interpreter is already available.

IMHO, bundling python binaries into your script would not only make your package much bigger, it would in fact make your script less portable as the binaries would be compiled for the specific OS and glibc. If someone wanted to use the script on a different (linux) OS/architecture it would not be possible unless you provided a package for that version.

Comments

[terdon]

Define "portable". Please edit your question and explain what systems this should work on. Will they all be Linux? Will they all be CentOS? All CentOS7?

[Yuusou]

If all servers are running CentOS, you may want to look at CentOS's "The Software Collections." wiki.centos.org/AdditionalResources/Repositories/SCL More specifically for SCL and Python27: softwarecollections.org/en/scls/rhscl/python27

[lese]

Thank you for the hint, I've updated the question, all the servers will be GNU/Linux, most of them will be CentOS, some of them Debian, or Debian based

[wjv]

virtualenv is a great tool, of course, but it doesn’t really add anything that aids portability across servers, which is what the original question asked. Wouldn’t it be better to point the asker at pip freeze, or for the more complicated case (dependencies on external libraries, etc.) at containerisation tools like Docker?

[nagu]

hmm, interesting. i'm unaware of pip freeze, I will need take a look. Docker adds another whole layer of complexity and additional dependencies which have to be managed. For the script that OP is writing (migrate mysql -> postres), it may be too much overhead to introduce docker, in my opinion, considering how to setup the connectivity from the container to mysql and postgress DBs (which themselves may not be containerized). YMMV.

[Bryce Guinta]

The python binary included in a virtualenv from an installed system is dynamically linked to the libpython shared object: ldd $(which python) | grep python returns something similar to libpython2.7.so.1.0 => /lib64/libpython2.7.so.1.0 (0x00007fe071dc9000)