Jenkinsfile and Python virtualenv

Question

I am trying to setup a project that uses the shiny new Jenkins pipelines, more specifically a multibranch project.

I have a Jenkinsfile created in a test branch as below:

node {
    stage 'Preparing VirtualEnv'
    if (!fileExists('.env')){
        echo 'Creating virtualenv ...'
        sh 'virtualenv --no-site-packages .env'
    }
    sh '. .env/bin/activate'
    sh 'ls -all'
    if (fileExists('requirements/preinstall.txt')){
        sh 'pip install -r requirements/preinstall.txt'
    }
    sh 'pip install -r requirements/test.txt'
    stage 'Unittests'
    sh './manage.py test --noinput'
}

It's worth noting that preinstall.txt will update pip.

I am getting error as below:

OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/pip'

Looks like it's trying to update pip in global env instead of inside virtualenv, and looks like each sh step is on its own context, how do I make them to execute within the same context?

Solution 1

What you are trying to do will not work. Every time you call the sh command, jenkins will create a new shell.

This means that if you use .env/bin/activate in a sh it will be only sourced in that shell session. The result is that in a new sh command you have to source the file again (if you take a closer look at the console output you will see that Jenkins will actually create temporary shell files each time you run the command.

So you should either source the .env/bin/activate file at the beginning of each shell command (you can use triple quotes for multiline strings), like so

if (fileExists('requirements/preinstall.txt')) {
    sh """
    . .env/bin/activate
    pip install -r requirements/preinstall.txt
    """
}
...
sh """
. .env/bin/activate
pip install -r requirements/test.txt
"""
}
stage("Unittests") {
    sh """
    . .env/bin/activate
    ./manage.py test --noinput
    """
}

or run it all in one shell

sh """
. .env/bin/activate
if [[ -f requirements/preinstall.txt ]]; then
    pip install -r requirements/preinstall.txt
fi
pip install -r requirements/test.txt
./manage.py test --noinput
"""

Solution 2

Like Rik posted, virtualenvs don't work well within the Jenkins Pipeline Environment, since a new shell is created for each command.

I created a plugin that makes this process a little less painful, which can be found here: https://wiki.jenkins.io/display/JENKINS/Pyenv+Pipeline+Plugin. It essentially just wraps each call in a way that activates the virtualenv prior to running the command. This in itself is tricky, as some methods of running multiple commands inline are split into two separate commands by Jenkins, causing the activated virtualenv no longer to apply.