Set up virtualenv using a requirements.txt generated by conda

Question

I'm setting up a python project, using an Anaconda virtual environment. I'm generating a requirements.txt so other people can easily set up their own virtual environment for the project.

I was wondering though, when other developers want to contribute to the project, but want to use virtualenv instead of Anaconda, can they do that?

I tried the following:

• I set up an empty project in an Anaconda environment and installed the aiohttp module. Then conda list --export > requirements.txt generates the following:

  # This file may be used to create an environment using:
  # $ conda create --name <env> --file <this file>
  # platform: win-64
  aiohttp=2.3.9=py36_0
  async-timeout=2.0.0=py36hc3e01a3_0
  certifi=2018.1.18=py36_0
  chardet=3.0.4=py36h420ce6e_1
  multidict=3.3.2=py36h72bac45_0
  pip=9.0.1=py36h226ae91_4
  python=3.6.4=h6538335_1
  setuptools=38.4.0=py36_0
  vc=14=h0510ff6_3
  vs2015_runtime=14.0.25123=3
  wheel=0.30.0=py36h6c3ec14_1
  wincertstore=0.2=py36h7fe50ca_0
  yarl=0.14.2=py36h27d1bf2_0
  

• I set up an empty project in a virtualenv environment and installed the aiohttp module there too. pip freeze > requirements.txt then generates:

  aiohttp==3.0.1
  async-timeout==2.0.0
  attrs==17.4.0
  chardet==3.0.4
  idna==2.6
  idna-ssl==1.0.0
  multidict==4.1.0
  yarl==1.1.0
  

So apparently both outputs are different, and my theory is: once I generate my requirements.txt with conda on my project, other developers can't choose virtualenv instead - at least not if they're not prepared to install a long list requirements by hand (it will be more than just the aiohttp module of course).

A first sight, importing the conda-generated requirements.txt in a project on virtualenv (pip install -r requirements-conda.txt) throws this error:

Invalid requirement: 'aiohttp=2.3.9=py36_0'
= is not a valid operator. Did you mean == ?

Am I right when I think that if developers would like to do this, they would need to programmatically change the package list to the format that virtualenv understands, or they would have to import all packages manually? Meaning that I impose them to choose conda as virtual environment as well if they want to save themselves some extra work?

Solution

I'm setting up a python project, using an Anaconda virtual environment. I was wondering though, when other developers want to contribute to the project, but want to use virtualenv instead of Anaconda, can they do that?

Yes - in fact this is how many of my projects are structured. To accomplish what you're looking for, here is a simple directory that we'll use as reference:

.
├── README.md
├── app
│   ├── __init__.py
│   ├── static
│   ├── templates
├── migrations
├── app.py
├── environment.yml
├── requirements.txt

In the project directory above, we have both environment.yml (for Conda users) and requirements.txt (for pip).

environment.yml

So apparently both outputs are different, and my theory is: once I generate my requirements.txt with conda on my project, other developers can't choose virtualenv instead - at least not if they're not prepared to install a long list requirements by hand (it will be more than just the aiohttp module of course).

To overcome this, we are using two different environment files, each in their own distinct format allowing for other contributors to pick the one they prefer. If Adam uses Conda to manage his environments, then all he need to do create his Conda from the environment.yml file:

conda env create -f environment.yml

The first line of the yml file sets the new environment's name. This is how we create the Conda-flavored environment file. Activate your environment (source activate or conda activate) then:

conda env export > environment.yml

In fact, because the environment file created by the conda env export command handles both the environment's pip packages and conda packages, we don't even need to have two distinct processes to create this file. conda env export will export all packages within your environment regardless of the channel they're installed from. It will have a record of this in environment.yml as well:

name: awesomeflask
channels:
- anaconda
- conda-forge
- defaults
dependencies:
- appnope=0.1.0=py36hf537a9a_0
- backcall=0.1.0=py36_0
- cffi=1.11.5=py36h6174b99_1
- decorator=4.3.0=py36_0
- ...

requirements.txt

Am I right when I think that if developers would like to do this, they would need to programmatically change the package list to the format that virtualenv understands, or they would have to import all packages manually? Meaning that I impose them to choose conda as virtual environment as well if they want to save themselves some extra work?

The recommended (and conventional) way to _change to the format that pip understands is through requirements.txt. Activate your environment (source activate or conda activate) then:

pip freeze > requirements.txt

Say Eve, one of the project contributor want to create an identical virtual environment from requirements.txt, she can either:

# using pip
pip install -r requirements.txt

# using Conda
conda create --name <env_name> --file requirements.txt

A full discussion is beyond the scope of this answer, but similar questions are worth a read.

An example of requirements.txt:

alembic==0.9.9
altair==2.2.2
appnope==0.1.0
arrow==0.12.1
asn1crypto==0.24.0
astroid==2.0.2
backcall==0.1.0
...

Tips: always create requirements.txt

In general, even on a project where all members are Conda users, I still recommend creating both the environment.yml (for the contributors) as well as the requirements.txt environment file. I also recommend having both these environment files tracked by version control early on (ideally from the beginning). This has many benefits, among them being the fact that it simplifies your debugging process and your deployment process later on.

A specific example that spring to mind is that of Azure App Service. Say you have a Django / Flask app, and want to deploy the app to a Linux server using Azure App Service with git deployment enabled:

az group create --name myResourceGroup --location "Southeast Asia"
az webapp create --resource-group myResourceGroup --plan myServicePlan

The service will look for two files, one being application.py and another being requirements.txt. You absolutely need both of these file (even if they're blank files) for the automation to work. This varies by cloud infrastructure / providers of course, but I find that having requirements.txt in our project generally saves us a lot of trouble down the road and worth the initial set-up overhead.

If your code is on GitHub, having requirements.txt also give you extra peace of mind by having its vulnerability detection pick up on any issue before alerting you / repo owner. That's a lot of great value for free, on the merit of having this extra dependency file (small price to pay).

This is as easy as making sure that every time a new dependency is installed, we run both the conda env export and pip freeze > requirements.txt command.

Comments

[darthbith]

The problem is not between conda and virtualenv, it is between conda and pip. And yes, the format between a conda environment.yml file and pip requirements.txt file are different. I'm not aware of any automatic converters between them.

[Sander Vanden Hautte]

Oh, OK, conda/pip as root issue makes sense.

[Sander Vanden Hautte]

Very thorough answer, I'll accept it. One minor problem I have with this approach is that a conda user adds a bunch of dependencies an checks in a changed environment.yml, a venv user will have troubles and will need to resolve his/her requirements.txt first, based on errors while running the project or warnings from his/her IDE. A single dependencies file would be better, but since that is not possible for now, this is the best possible approach, I agree.

[Luise]

shouldn't the command for # using Conda be conda env create --name <env_name> --file requirements.txt?

[Venkatesh Dharavath]

@SanderVandenHautte, I'm hitting the same issue. Trying to create a virtualenv from the conda env. I got the requirements.txt but there are stuff in that file that are related to conda only. I removed some of them manually, still the installation fails. There should a way. Please put it here if you have any workaround for this.

[onlyphantom]

To be honest @VenkateshDharavath there is nothing stopping two or more users who exclusively use pip to mess up the dependencies file the same way. User A uses venv. User B uses venv. User B added a bunch of lines that are not really dependencies to the project. User A remove them manually. So this isn't something that would be solved by removing conda from the workflow. It's also why code reviews and pull requests are essential

[Venkatesh Dharavath]

Yeah, I agree @onlyphantom