Using multiple Python engines (32Bit/64bit and 2.7/3.5)

Question

I would like to use Python for scientific applications and after some research decided that I will use Anaconda as it comes bundled with loads of packages and add new modules using conda install through the cmd is easy.

I prefer to use the 64 bit version for better RAM use and efficiency but 32bit version is needed as well because some libraries are 32bit. Similarly, I prefer to use Python 3.5 as that is the future and the way things go. But loads of libraries are still 2.7 which means I need both.

I have to install 4 versions of Anaconda (64bit 2.7, 64bit 3.5, 32bit 2.7, 64bit 3.5). Each version is about 380MB. I am aiming to use Jupyter notebook and Spyder as the IDE. I had to switch between versions when required. I had conflicting libraries, path issues and all sorts of weird problems.

So, I am planning to do a clean install from scratch. I would like to know if there is a more sensible way to handle this. I use Windows 7 64 bit for now if that matters.

Solution 1

Make sure to set the right environmental variables (https://github.com/conda/conda/issues/1744)

Create a new environment for 32bit Python 2.7:

set CONDA_FORCE_32BIT=1
conda create -n py27_32 python=2.7

Activate it:

set CONDA_FORCE_32BIT=1
activate py27_32

Deactivate it:

deactivate py27_32

Create one for 64 bit Python 3.5:

set CONDA_FORCE_32BIT=
conda create -n py35_64 python=3.5

Activate it:

set CONDA_FORCE_32BIT=
activate py35_64

The best would be to write the activation commands in a batch file so that you have to type only one command and cannot forget to set the right 32/64 bit flag.

UPDATE

You don't need to install a full Anaconda distribution for this. Miniconda is enough:

These Miniconda installers contain the conda package manager and Python. Once Miniconda is installed, you can use the conda command to install any other packages and create environments, etc. ...

There are two variants of the installer: Miniconda is Python 2 based and Miniconda3 is Python 3 based. Note that the choice of which Miniconda is installed only affects the root environment. Regardless of which version of Miniconda you install, you can still install both Python 2.x and Python 3.x environments.

I would recommend you to use Miniconda3 64-bit as your root environment.

You can always install a full Anaconda later with:

conda install anaconda

Note that it might downgrade some of your previously install packages in your active environment.

Solution 2

Setting the Subdirectory Constraint

Conda has a configuration variable subdir that can be used to constrain package searching to platforms (e.g., win-32). I think the simplest procedure is to create the empty env, set it's subdir, then proceed with the (constrained) installations. For example,

win-32, Python 2.7

conda create -n py27_32
conda activate py27_32
conda config --env --set subdir win-32
conda install python=2.7

win-64, Python 3.7

conda create -n py37_64
conda activate py37_64
conda config --env --set subdir win-64
conda install python=3.7

Alternatively, if you need to, for example, create an environment from a YAML file, but want a win-32 platform, one can use the CONDA_SUBDIR environment variable:

set CONDA_SUBDIR=win-32
conda env create -f env.yaml -n my_env_32
set CONDA_SUBDIR=
conda activate my_env_32
conda config --env --set subdir win-32

The nice thing about this procedure is the variable will now always be set whenever activating the env, so future changes to the env will remain within the specified subdirectory.

---

Ad Hoc Constraints

It is also possible to specify the platform in the --channel|-c argument:

conda install -c defaults/win-32 --override-channels python=3.7

Here the --override-channels is required to ensure that only the provided channel(s) and subdirectory (win-32) is used.

However, setting the subdir on the whole env is likely a more reliable practice.

---

YAML Constraints

It is also possible to use subdir specifications in a YAML environment definition. However, this is less reliable (see below and comments). For example,

py37_win32.yaml

name: py37_win32
channels:
 - defaults/win-32
dependencies:
 - python=3.7

@Bicudo has tried this and confirms it works, but notes that it does not set any environment-specific constraints on future updates to the environment. Additionally, @Geeocode pointed out that the default subdir can still leak in, which is likely due to conda env create still having access to the global channels configuration during solving (there is no equivalent --override-channels flag for conda env create). Hence, it would be good practice to still set the subdir before and after environment creation, e.g.,

set CONDA_SUBDIR=win-32
conda env create -f py37_win32.yaml
set CONDA_SUBDIR=
conda activate py37_win32
conda config --env --set subdir win-32

Alternatively, beginning with Conda v4.9, one can also specify environment variables as part of the YAML. That is, one can effectively define an environment's CONDA_SUBDIR value at environment creation:

py37_win32.yaml

name: py37_win32
channels:
 - defaults/win-32
dependencies:
 - python=3.7
variables:
  CONDA_SUBDIR: win-32

Solution 3

I just wanted to add to Mike Mullers answer, as I also wanted my IPython to switch between 32 bit and 64 bit.

After setting up the 32bit or 64bit environment. Use the following commands

pip install ipykernel

to install ipykernel on this env. Then assign it with:

python -m ipykernel install --user --name myenv --display-name "Python (myenv)"

here myenv is the name of your new environment. See this page here for further details on switching kernels - http://ipython.readthedocs.io/en/stable/install/kernel_install.html

Solution 4

(now in conda win64 - python64 activate env)

set CONDA_SUBDIR=win-32
conda install python

you will see

The following packages will be SUPERSEDED by a higher-priority channel:

ca-certificates anaconda/pkgs/main/win-64::ca-certifi~ --> anaconda/pkgs/main/win-32::ca-certificates-2021.7.5-h9f7ea03_1
openssl anaconda/pkgs/main/win-64::openssl-1.~ --> anaconda/pkgs/main/win-32::openssl-1.1.1k-hc431981_0 python
anaconda/pkgs/main/win-64::python-3.9~ --> anaconda/pkgs/main/win-32::python-3.9.5-h53c7b84_3

Proceed ([y]/n)?

just type "y"

this setting is saved in file "\anaconda\envs\ you env \ .condarc"

View more solutions

Comments

[cel]

You do not need a separate conda installation for each python version. Instead you may want to familiarize yourself with the concept of conda's environments. Nowadays it should be possible to get a 64bit version from almost any library. If this is not the case (which I would definitely check) , you will have to maintain two separate anaconda versions, which make things a little more complicated

[mindlessgreen]

@cel That's cool! conda environments is probably what I am looking for. A quick look at conda.pydata.org/docs/py2or3.html shows that I can have multiple environments for 2.7, 3,5 etc with separate libraries. But I am not sure 32bit/64bit issue can be assigned into environments. Anyway, It would be nice if you could add your comment as an answer.

[simonzack]

Why bother with anaconda? Vanilla python has everything anaconda has, and is a lot more flexible, you have things like pyenv to easily install multiple environments and fork off them.

[mindlessgreen]

nice. Do I just install any one version of anaconda (say 64 bit 2.7) and set up environments? Or do I need to have multiple versions of anaconda installed for the environments to work?

[Mike Müller]

Added a paragraph about Miniconda to my answer. This should answer your question.

[Wolfgang Ulmer]

Thanks a lot! Didn't know about CONDA_FORCE_32BIT until now.

[Daniel Schreij]

Thank you! I have been looking for this for ages, and the only thing I could repeatedly find was "This is not possible" ...

[mnut]

I tried to install a 32 bits python env in anaconda 64 bits under windows : qtconsole, scipy, jupyter were not working (alhough they claimed to be installed correctly). When I installed a new version of anaconda 32 bits, everything worked ok.

[poleguy]

This works for trivial examples, but conda still seems to install a 64 bit version of numpy, so import numpy gives a "not a valid Win32 application" error.

[poleguy]

It seems CONDA_FORCE_32BIT is going away, and this isn't easy to get working currently: github.com/conda/conda/issues/1744

[user32882]

@poleguy its 2019 now and I was still able to use it. I don't see why it should go away. It's extremely useful and allows for one miniconda installation to give you all possibilities: python2 32 bit, python2 64 bit, python3 32 bit and python3 43 bit

[merv]

@user32882 even if it does go away, setting a subdirectory constraint is sufficiently general to subsume the functionality CONDA_FORCE_32BIT provided.

[Bicudo]

The last option works but not setting the subdir leads to problems with pip dependencies and making any package modifications later (conda will try to supersede installations with 64bit versions).

[merv]

@Bicudo thanks giving feedback. One clarification, when making “package modifications”, do you mean using conda install? Or have you tried a YAML-only approach, i.e., edit the YAML and then use conda env update? Also, I’m surprised pip would be an issue. As long as the pip module was included in the original YAML, it should be architecture-specific and appropriately matched with the installed Python.

[Bicudo]

Yes, conda install caused the mentioned issue here, but I haven't tried conda env update, that might work better indeed. With pip, I'm not completely sure and you are probably right, perhaps the issue was indeed only with conda. I see you updated your answer, nice :) so I probably can delete my comment not to cause any misunderstanding on the pip matter.

[Geeocode]

Just tried and without CONDA_FORCE_32BIT it didn't work, because it installed the 64bit Python.

[merv]

@Geeocode thanks for the feedback! Which method did you try? It is possible that without the —override-channels argument, the default subdirs in one’s global configuration are still accessible.

[Geeocode]

I tried the yaml version, as I wanted to clone an existing env. It seemed that it tend to choose the 32 versions except the Python version itself.

[merv]

@Geeocode okay, that was what I suspected. There is no --override-channels option with conda env create, so it likely is considering the other channels in your global config. I suspect that setting CONDA_CHANNEL_PRIORITY='strict' could also do the trick, but may still not be as reliable as CONDA_FORCE_32BIT.

[Geeocode]

Tried CONDA_CHANNEL_PRIORITY='strict' still 64bit Python. Then tried inserting into the variables section of yaml CONDA_FORCE_32BIT: 1, CONDA_FORCE_32BIT: True, CONDA_FORCE_32BIT: true also, still no success. So far the only working solution was the direct env set i.e. set CONDA_FORCE_32BIT=1.

[merv]

@Geeocode the variables section only get used when the environment is activated, so no surprise that wouldn't work. Alternatively, set CONDA_SUBDIR=win-32 in the shell prior to conda env create should still work, as shown in the first section of the answer.

[Geeocode]

I confirm, that the set CONDA_SUBDIR=win-32 finally works as expected. Thank you for your efforts.