How to install my own python module (package) via conda and watch its changes

python module package anaconda miniconda

45,627

Solution 1

If you install the conda build package (and you have a package, not just a script), you can install in "editable" mode:

conda develop .

(running from the directory with your script). This is very similar to the "editable" mode from pip

pip install -e .

Either approach lets you uninstall packages with either

conda develop -u .

pip uninstall .

If you just have a script (not a package), you can edit or set the PYTHONPATH environment variable to include the directory with the script.

Solution 2

While the previous answers are doing what I need, I just want to show what I will be using instead. Since it was my plan to learn about conda packages anyway...

0. Good sources

1. Create a python package template for conda using cookiecutter

conda install -c conda-forge cookiecutter

Now change to the directory where you want to initialize your package, then do:

cookiecutter https://github.com/conda/cookiecutter-conda-python.git

This will ask for some basic information about the package that you want to create. Then change into your repo

cd myrepo

2. Build your package

make sure conda-build is installed, if not run

conda install conda-build

Make sure to set the CONDA_BLD_PATH as mentioned in anaconda - using a different conda-build root directory - Stack Overflow. This will be the directory where you can find your packages, then run:

conda build conda.recipe

to build your package and clean up after you with

conda build purge

3. Set up your own local channel (no uploading to anaconda.org)

Read

for help.

Index each platform. Maybe someone can confirm that this step is not needed, as my builds already contain the repodata.json. Otherwise:

conda index D:\CODE\condamychannel\win-64

Test if the package can be found with

conda search -c file:///D:\CODE\condamychannel --override-channels mypackage

or add the channel to the config directly (per environment)

conda config --add channels file:///D:\CODE\condamychannel

4. Install (and update) the package

activate myenv

and

conda install mypackage

Once I change mypackage, I give it a new version number in meta.yaml and setup.py and build the package with conda build conda.recipe. Updating is simply

conda update mypackage

See if your package works:

python
>>> import cli from mypackage
>>> cli.cli()
CLI template

This may not be the optimal way, but I could not find a tutorial that contains all the steps I outlined above.

Solution 3

I had a wrapper function that I had to call multiple times on different scripts.So, I copied that file wrappers.py to the Anaconda site-packages folder. On my computer this was at: C:\ProgramData\Anaconda3\Lib\site-packages. Then, whenever I needed wrappers.py, I'd just import it in my scripts, like this:

import wrappers

If you want to make sure that the import was successful, you could either select Anaconda as your dev environment in your IDE and then invoke the Intellisense after import: from wrappers import (intellisense suggestions). Or you could also use IDLE:

>>> import wrappers
>>>(if you get this prompt, this means the import was successful).

Conda integration is only possible if you create custom channels. This is because conda searches for packages in the following locations(based on your OS) and to tell conda to search for your module at certain location(s), you must install conda-build to create a custom channel:

- https://repo.continuum.io/pkgs/main/win-64
- https://repo.continuum.io/pkgs/main/noarch
- https://repo.continuum.io/pkgs/free/win-64
- https://repo.continuum.io/pkgs/free/noarch
- https://repo.continuum.io/pkgs/r/win-64
- https://repo.continuum.io/pkgs/r/noarch
- https://repo.continuum.io/pkgs/pro/win-64
- https://repo.continuum.io/pkgs/pro/noarch
- https://repo.continuum.io/pkgs/msys2/win-64
- https://repo.continuum.io/pkgs/msys2/noarch

If you want to avoid creating a repo/channel, then a simple import as above should do the job. In addition, if you make any changes to your module and save it, you will always have the latest import in your scripts.

45,627

Author by

Michael

Geospatial Data Scientist

Updated on July 05, 2022

Comments

Michael almost 2 years

I have a file mysql.py, which I use in almost all of my projects. Since I do not want to copy and paste the same file into each of these projects I wrote a module - possibly a package in the future.

Question

How do I add a local module to my conda environment, and automatically update or watch it when I change something in mysql.py? How to do the same for a package?

I would like to avoid to set up a local channel/repository and just reference to the folder with mysql.py.
Charlie Parker over 4 years

why did you choose to not use conda develop or pip -e?
Charlie Parker over 4 years

how do you know when to you pip -e vs conda develop?
darthbith over 4 years

If you're using conda, you should use conda develop. The only small problem I ran into with conda develop was that it would not install console_scripts, whereas pip install -e would. This may have been fixed in conda in the meantime though.
Charlie Parker over 4 years

how do you check if things were installed? I tried running conda list but nothing familiar to my project/packages seem to show up.
darthbith over 4 years

It should show up in the output of conda list. Do you have a setup.py file properly configured? Does pip install -e work?
Michele Piccolini about 4 years

@CharlieParker See jakevdp.github.io/blog/2016/08/25/… for pip vs conda.
Mint over 3 years

After installing conda build, using the conda develop option worked but the package does not show up in conda list. It shows up in pip list and imports fine. I know conda list shows somewhat different items than pip list, but I would expect it to show my package like pip list does. Anyone know why it doesn't? When installed with pip install -e . the package shows up in both pip list and conda list
Peter almost 3 years

Why doing it the easy way if it can be done so much more cumbersome!
E. Davis almost 3 years

See http://evantilton.com/guides/anacondapythonpath/ for a simple method to programmatically change PYTHONPATH for a particular conda environment.
Glxblt76 over 2 years

I tried to follow the instructions here but it said at "cookiecutter github.com/conda/cookiecutter-conda-python.git" that git is not installed. I tried conda install git but I got the same message.
Martin Hepp over 2 years

I actually like this approach better than conda develop because you can also include your own modules in environment YAML files and more. Also, upgrading to a fully-fledged public module is straightforward this way.
Martin Hepp over 2 years

Addendum: Using pip in conda environments works, but there are lots of reports of potential issues with this. It seems to work okay for a new environment, but updating may create inconsistencies. Personally I try to avoid pip inside a conda environment.
Martin Hepp over 2 years

Note that conda develop is not actively maintained and there is a request to deprecate or remove it, see github.com/conda/conda-build/issues/4251. So **it is not advisable to use conda develop in 2021.
Martin Hepp over 2 years

Addendum 2: There is a request to deprecate conda develop from conda, see github.com/conda/conda-build/issues/4251
Justin P. about 2 years

Adding to @MartinHepp: The recommendation seems to remain: use python setup.py develop or pip install -e instead, see github.com/conda/conda-build/issues/1992.
eric about 2 years

@JustinP. that isse at conda is basically a bunch of developers who are super confused about what to do: it seems conda should provide a canonical source of guidance about this. In the meantime, I will try using pip install -e . until it doesn't work.