Failing to get PyCharm to work with remote interpreter on docker

Question

When I add a remote interpreter from one of my docker-compose, it doesn't seem to succeed and doesn't show any packages in the dialog. When I add an interpreter to the debugger it says:

python packaging tools not found.

Then if i click on install packaging tools, error displayed:

ERROR: for dockeryard_pycharm_helpers_1
Cannot start service pycharm_helpers: network not found

Starting dockeryard_postgres_1 ...  
Starting dockeryard_nginx_1    ...  
Starting dockeryard_redis_1    ... 
Starting dockeryard_postgres_1 ... 
Starting dockeryard_nginx_1 ...  
Starting dockeryard_pycharm_helpers_1  
Starting dockeryard_redis_1  
Starting dockeryard_worker_1 ... 
Starting dockeryard_worker_1
Starting dockeryard_pycharm_helpers_1
ERROR: for dockeryard_pycharm_helpers_1  Cannot start service     pycharm_helpers: network not found

ERROR: for pycharm_helpers  Cannot start service pycharm_helpers: network not found
[31m  
ERROR [0m: 

Note, this interpreter was already in use and I was able to connect remotely with PyCharm, but I have added and eventually removed a custom network to the container.

As explained in Configuring Remote Python Interpreters - "When a remote Python interpreter is added, at first the PyCharm helpers are copied to the remote host". And my guess something went wrong since network was updated in the docker-compose.

From what I understand from the error message, when PyCharm starts interpreter it tries to use/find that network c7b0cc277c94ba5f58f6e72dcbab1ba24794e72422e839a83ea6102d08c40452.

I don't see that network listed anywhere when I run:

$ docker network inspect dockeryard_default

So PyCharm stores it somewhere and not been updated with the change. I have tried to remove interpreter (using PyCharm dialog) and add it back - same result.

How can I get rid of this network and make PyCharm to be able to debug again?

Thanks.

Solution 1

Was having a near identical error and was able to get past it. I did two things though I'm uncertain as to which was the actual solution:

1. Made sure the mappings were correct under both (a) Preferences -> Project -> Project Interpreter -> Path mappings and (b) Run -> Edit Configurations -> <Your_Configuration> -> Path mappings
2. Removed/deleted any containers that looked to be related to PyCharm (believe this is more than likely what solved things).

Hope this helps. PyCharm docker-compose seems to work for some and be a real PITA for others.

One other note. I downgraded from PyCharm 2018 to 2017.3 as there's known docker bugs in 2018.

EDIT: And it would seem a docker-compose down from CLI reintroduces the error -_-

Solution 2

TLDR:

The {project_name}_pycharm_helpers_{pycharm_build_number} volume has been removed or is corrupted.

To repopulate it, run:

docker volume rm {project_name}_pycharm_helpers_{pycharm_build_number}
docker run -v {project_name}_pycharm_helpers_{pycharm_build_number}:/opt/.pycharm_helpers pycharm_helpers:{pycharm_build_number}

The pycharm_build_number can be found in the about section of your pycharm (mac OS: Pycharm > About)

Long story

I struggled a lot with PyCharm suddenly not finding the helpers any more or any related bugs, sometimes because I was clearing my containers or volumes. For instance, running

docker rm -f `docker container ps -aq`
docker volume rm $(docker volume ls -q)

will almost surely get pycharm into troubles.

AFAIK about how PyCharm works, there is:

• a PyCharm base image named pycharm_helpers with tag corresponding to your pycharm build number (for example: PY-202.7660.27)
• the first time you create docker related things, PyCharm creates volumes that get data from this image for later use in your containers. For instance, after a first attempt at running a remote docker-compose interpreter, I see the newly created myproject_pycharm_helpers_PY-202.7660.27 volume when doing docker volume ls.
• when running the docker interpreter, PyCharm adds this volume into the /opt/.pycharm_helpers directory by adding at some point a -v myproject_pycharm_helpers_PY-202.7660.27:/opt/.pycharm_helpers to your command. For instance using docker-compose, you can see the addition of the -f /Users/clementwalter/Library/Caches/JetBrains/PyCharm2020.2/tmp/docker-compose.override.1508.yml and when you actually look into this file you see:

version: "3.8"
services:
  local:
    command:
    - "python"
    - "/opt/.pycharm_helpers/pydev/pydevconsole.py"
    - "--mode=server"
    - "--port=55824"
    entrypoint: ""
    environment:
      PYCHARM_MATPLOTLIB_INTERACTIVE: "true"
      PYTHONPATH: "/opt/project:/opt/.pycharm_helpers/pycharm_matplotlib_backend:/opt/.pycharm_helpers/pycharm_display:/opt/.pycharm_helpers/third_party/thriftpy:/opt/.pycharm_helpers/pydev"
      PYTHONUNBUFFERED: "1"
      PYTHONIOENCODING: "UTF-8"
      PYCHARM_MATPLOTLIB_INDEX: "0"
      PYCHARM_HOSTED: "1"
      PYCHARM_DISPLAY_PORT: "63342"
      IPYTHONENABLE: "True"
    volumes:
    - "/Users/clementwalter/Documents/myproject:/opt/project:rw"
    - "pycharm_helpers_PY-202.7660.27:/opt/.pycharm_helpers"
    working_dir: "/opt/project"
volumes:
  pycharm_helpers_PY-202.7660.27: {}

You get into troubles when this volume is not correctly populated anymore. Fortunately the docker volume documentation has a section "Populate a volume using a container" which is exactly what PyCharm does under the hood.

For the record you can check the content of the pycharm_helpers image:

$ docker run -it pycharm_helpers:PY-202.7660.27 sh
/opt/.pycharm_helpers #

you end up into the pycharm_helpers directory and find all the helpers here:

/opt/.pycharm_helpers # ls -la
total 5568
drwxr-xr-x   21 root     root          4096 Dec 17 16:38 .
drwxr-xr-x    1 root     root          4096 Dec 17 11:07 ..
-rw-r--r--    1 root     root           274 Dec 17 11:07 Dockerfile
drwxr-xr-x    5 root     root          4096 Dec 17 16:38 MathJax
-rw-r--r--    1 root     root          2526 Sep 16 11:14 check_all_test_suite.py
-rw-r--r--    1 root     root          3194 Sep 16 11:14 conda_packaging_tool.py
drwxr-xr-x    2 root     root          4096 Dec 17 16:38 coverage_runner
drwxr-xr-x    3 root     root          4096 Dec 17 16:38 coveragepy
-rw-r--r--    1 root     root         11586 Sep 16 11:14 docstring_formatter.py
drwxr-xr-x    4 root     root          4096 Dec 17 16:38 epydoc
-rw-r--r--    1 root     root           519 Sep 16 11:14 extra_syspath.py
drwxr-xr-x    3 root     root          4096 Dec 17 16:38 generator3
-rw-r--r--    1 root     root             8 Sep 16 11:14 icon-robots.txt
-rw-r--r--    1 root     root          3950 Sep 16 11:14 packaging_tool.py
-rw-r--r--    1 root     root       1490666 Sep 16 11:14 pip-20.1.1-py2.py3-none-any.whl
drwxr-xr-x    2 root     root          4096 Dec 17 16:38 pockets
drwxr-xr-x    3 root     root          4096 Dec 17 16:38 profiler
-rw-r--r--    1 root     root           863 Sep 16 11:14 py2ipnb_converter.py
drwxr-xr-x    3 root     root          4096 Dec 17 16:38 py2only
drwxr-xr-x    3 root     root          4096 Dec 17 16:38 py3only
drwxr-xr-x    7 root     root          4096 Dec 17 16:38 pycharm
drwxr-xr-x    4 root     root          4096 Dec 17 16:38 pycharm_display
drwxr-xr-x    3 root     root          4096 Dec 17 16:38 pycharm_matplotlib_backend
-rw-r--r--    1 root     root        103414 Sep 16 11:14 pycodestyle.py
drwxr-xr-x   24 root     root          4096 Dec 17 16:38 pydev
drwxr-xr-x    9 root     root          4096 Dec 17 16:38 python-skeletons
drwxr-xr-x    2 root     root          4096 Dec 17 16:38 rest_runners
-rw-r--r--    1 root     root        583493 Sep 16 11:14 setuptools-44.1.1-py2.py3-none-any.whl
-rw-r--r--    1 root     root         29664 Sep 16 11:14 six.py
drwxr-xr-x    3 root     root          4096 Dec 17 16:38 sphinxcontrib
-rw-r--r--    1 root     root           128 Sep 16 11:14 syspath.py
drwxr-xr-x    3 root     root          4096 Dec 17 16:38 third_party
drwxr-xr-x    3 root     root          4096 Dec 17 16:38 tools
drwxr-xr-x    5 root     root          4096 Dec 17 16:38 typeshed
-rw-r--r--    1 root     root       3354133 Sep 16 11:14 virtualenv-16.7.10-py2.py3-none-any.whl

• to make these helpers available again, following the docker documentation, you have to fix the volume. To do so:

docker rm {project_name}_pycharm_helpers_{pycharm_build}
docker run -v {project_name}_pycharm_helpers_{pycharm_build}:"/opt/.pycharm_helpers" pycharm_helpers:{tag}

et voilà

Solution 3

If you're still seeing this in PyCharm 2020.2 then do this:

• close PyCharm
• try @peterc's suggestion:

docker ps -a | grep -i pycharm | awk '{print $1}' | xargs docker rm

• launch PyCharm again

Solution 4

The option invalidate cache -> Clear downloaded shared indexes will also repopulate the Pycharm volumes. (At least in 2021.1)

View more solutions

Comments

[peterc]

I did the second of these things and it fixed my problem. docker ps -a | grep -i pycharm | awk '{print $1}' | xargs docker rm

[Aisuko]

Remove the container named 'pycharm_helpers' is work for me.

[Milso]

Using docker-compose v2 instead of v3 for docker-compose interpreter is much more stable for me.

[Neil]

To get this to work, I deleted all my containers and images.

[Andrea Thacker]

@peterc needs to post his comment as the answer, that was the solution for me.

[21rw]

I found that invalidate cache and restart also repopulated the volumes.

[Karen]

thanks @ClementWalter, brilliant!

[alairock]

Worked for me still in Jan 2022, thanks!

[Gnut]

@21rw is correct. Run docker volume prune, delete all pycharm related container, then invalidate caches (tick the options clear VCS logs and clear downloaded shared index) works for me.