Jupyter notebook, how to execute system shell commands in the right conda environnment?

Question

I'm currently experiencing some troubles with jupyter notebook and system shell commands. I use nb_conda_kernels to be able to access all of my conda environment from a jupyter notebook launched in base environment, and this works perfectly in most of my use cases. For simplicity sake, let's assume I have 2 environments, the base one, and one named work_env. I launch jupyter notebook in the base environment, and select the work_env kernel upon opening the notebook I'm working on.

Today I came across this line:

! pip install kaggle --upgrade

upon execution of the cell (with the work_env kernel correctly activated), pip installed the kaggle package in my base environment. The intended result was to install this package in my work_env. Any ideas on how to make shell commands execute in the "right" environment from jupyter notebook?

Solution

Try specifying the current python interpreter.

import sys

!$sys.executable -m pip install kaggle --upgrade

sys.executable returns the path to the python interpreter you are currently running. $ passes that variable to your terminal (! runs the command on the terminal).

Aliases expand Python variables just like system calls using ! or !! do: all expressions prefixed with ‘$’ get expanded. For details of the semantic rules, see PEP-215

from https://ipython.org/ipython-doc/3/interactive/magics.html

-m is used to run a library module (pip in this case) as a script (check python -h). Running pip as a script guarantees that you are using the pip linked to the current python interpreter rather than the one specified by your system variables.

So, in this way you are sure that pip is installing dependencies on the very same python interpreter you are working on (which is installed in your current environment), this does the trick.