How can I gdb attach to a process running in a docker container?

Question

I have a long-running process in a docker container to which I would like to attach gdb to look at what threads are running and get stacktraces. I can attach to the process from the host, but I cannot resolve any symbols because the executable file is in a different location in the filesystem (it's in a docker-mounted volume) and the shared system libraries are all stuck in a docker filesystem image somewhere in /var/lib/docker.

I am able to generate a core file and use gdb to look at it by specifying the host's path to the executable file, but because the system libraries are all in the wrong places and are loaded in the wrong locations in the corefile, I get no information from that.

Do I have any options that I've overlooked?

Solution

You can attach to the process with gdb instance running in your container by attaching to the running container via lxc-attach.

Note: gdb has to be already installed in that container or you have to install it.

# find your container ID
sudo docker ps
# list of your containers - container ID is 1234567890
# find your full container ID
sudo docker ps --no-trunc -q| grep <short ID>
sudo lxc-attach -n <container long ID>

root@1234567890:/#
# optionally, you can install gdb now if it is not installed
# yum install gdb

root@1234567890:/# gdb
...
(gdb) attach 1

UPDATE 2017-04:

There is an easier workflow using docker exec now available (thanks to @42n4).

# find your container ID
sudo docker ps
# list of your containers - container ID is 1234567890
docker exec -i -t 1234567890 /bin/bash

root@1234567890:/#
# optionally, you can install gdb now if it is not installed
# yum install gdb

root@1234567890:/# gdb
...
(gdb) attach 1

Comments

[Evgen Bodunov]

Last docker uses "Id" instead of "ID". sudo docker inspect 1234567890 | grep '"Id"' | sed 's/[^0-9a-z]//g'

[42n4]

This gives all long ids: docker ps --no-trunc | awk '{print $1}'; but lxc-attach does not work now, maybe docker exec -it 'short id' /bin/bash?

[Jiri]

@42n4 docker ps --no-trunc -q| grep <short ID> is also possible

[Szczepan Hołyszewski]

That requires forking your development process to prepare a special image with BOTH your php application AND gdb. I am under the impression that this is SPECIFICALLY what the OP wants to avoid. They want to be able to use host gdb to attach to a containerized process.