Why doesn't a new Conda environment come with packages like numpy?

Question

I am going through the painful process of learning how to manage packages/ different (virtual) environments in Python/Anaconda. I was told that Anaconda is basically a python installation with all the packages I need (e.g. numpy, scipy, sci-kit learn etc).

However, when I create a new environment, none of these packages is readily available. I cannot import them when using PyCharm with the newly created environment. When I check the Pycharm project interpreter, or the anaconda navigator environments tab, It seems that indeed none of these packages are installed in my new environments. Why is this? It doesn't make sense to me to provide all these packages, but then not make them ready for use when creating new environments. Do I have to install all these packages manually in new env's or am I missing something?

Kindest regards, and thanks in advance.

Solution 1

The reason the default python environment doesn't come with numpy is because maybe you don't want numpy in the environment. Imagine writing an API (or general software package) where your users may or may not have access to numpy. You might want to run tests to make sure your software fails gracefully or has a pure python fallback if numpy is not installed on your user's machine. Conda environments provide this (insanely useful) benefit. Of course, the package in question doesn't have to be numpy. There are some more esoteric packages where this type of testing is useful.

Furthermore, you can create a conda environment with numpy pre-installed, or any other package you want pre-installed (just add them to the end of the conda create command):

conda create --name my-env-name numpy

Solution 2

Anaconda comes with available packages such as numpy, scipy, and sci-kit learn, but if you want to use them within your environment, you must:

1) Create the environment:

conda create --name new_env 

2) Activate the environment:

source activate new_env 

3) Install the desired package using conda install

conda install numpy

If you'd like to create a new environment that includes installations of all available Anaconda packages, see create anaconda python environment with all packages. You can include anaconda in the list of packages to install in the environment, which is a 'meta-package' meaning 'all the packages that come with the Anaconda installation'.

Solution 3

I don't know about "conda" environments but in general virtual environments are used to provide you a "unique" environment. This might include different packages, different environment variables etc.

The whole point of making a new virtual environment is to have a separate place where you can install all the binaries ( and other resources ) required for your project. If you have some pre-installed binaries in the environment, doesn't it defeat the purpose of creating one in the first place?

The fact that you can create multiple environments helps you to separate binaries that might be needed by one and not by the other.

For instance, if you are creating a project which requires numpy:1.1 but you have numpy:2.1 installed , then you have to change it. So basically, by not installing any other packages, they are not making assumptions about your project's requirements.

Comments

[roganjosh]

That isn't what the OP is asking

[Psychotechnopath]

This makes complete sense. However, what I do not understand, is what is the pre of using anaconda over say a normal python installation, if I still have to install these packages in a new env? I understand that conda is a very useful environment manager, but basically you can achieve the exact same with a normal python installation and pip. What is the advantage of Anaconda coming with all these packages pre-installed? Does it make them easier to install them for conda or something?

[Matt Messersmith]

@Psychotechnopath I'm not sure I entirely follow your question. You're not wrong in saying that you could achieve the same thing using pip, but it's just much more difficult to do. pip doesn't give you a good way of starting from a clean environment (no packages installed, just standard python). You can always manually uninstall and reinstall using pip for every package, but it's cumbersome (you'd have to create a list of packages or scrape them from pip freeze). Another adv of conda is being able to quickly switch envs that contain different packages. Is this what you're asking?

[Psychotechnopath]

To my understanding, Anaconda is a python installation+ some useful data-science packages, like numpy. However, when I create new envs with conda, these packages are not automatically installed, for the reason you mentioned in your answer. My question is: Why install anaconda then, when these packages are not readily available but only in the root/base directory of anaconda? To my understanding they are pre-installed but you can not use them when you create a new env. Or can you also just use the anaconda root/base directory as python interpreter, to enjoy all the packages being pre-installed?

[Matt Messersmith]

@Psychotechnopath The base Anaconda interpreter has the useful scientific computing packages preinstalled. You can also point your PATH to the Python interpreter that comes with Anaconda to get this benefit (Anaconda install does this automatically if you enter y when it asks on the prompt). You don't even have to use conda if you don't want to. Anaconda != conda (as confusing as that is)

[Psychotechnopath]

Thank you very much! Everything is completely clear to me now. It sounds confusing indeed, but Anaconda is just the python+packages installation, whereas conda is the environment managemant system (like venv).

[quantumbutterfly]

I thought the whole point of anaconda is that it comes with all these packages pre-installed. If you need to install them anyways in your environment, doesn't that defeat the whole point?