strace to monitor Dockerized application activity

Question

My goal is to monitor which ports are opened and closed by a multi-process application. My plan is to run the application in a Docker container, in order to isolate it, and then use strace to report the application activity.

I've tried with Apache server dockerized :

strace -f -o /tmp/docker.out docker run -D -P apache

I don't see any line in the report file that shows that the application accept a connection in a socket.

Can strace report the activity of processes inside the container?

Solution 1

The issue with your command+strace combination is that docker has a client/server model, and your docker run represents the client side of a REST API transaction to ask the docker daemon to run the Apache container on your behalf. Depending on how your client is configured, that container may not even run on the same system on which you type your docker run command.

However, to take the simplest case where the Docker client and daemon are on the same system, you can use ps find the PID of the running Apache server and use strace to join and trace the already-started process, as long as that is sufficient for your tracing needs.

Given I had to debug several early-start issues with "runc", the executor for containers in docker version 1.11 and above, I also created a small wrapper for docker-runc which strace's the container process from the start (from the outside system, so strace is not required in the container filesystem). You can find it here on GitHub, although fair warning that it is somewhat buggy for regular use as I believe the shell+strace invocation gets in the way of some signaling between containerd and the real docker-runc and associated processes. A more elegant solution might be to create a variant of runc which knows how to prepend the actual start of the contained process with an strace wrapper rather than intercepting the entire invocation of runc in an strace.

Solution 2

Take a look at the solution described at https://medium.com/@rothgar/how-to-debug-a-running-docker-container-from-a-separate-container-983f11740dc6 which tells you how to fire up a container with strace installed, which is in the same pid and network namespace as the container/process you wish to run strace against.

This is nice since it means you don't need to install strace in the container you wish to debug.

The guts of it is that when debugging a container (caddy in the example below) you run a docker container called strace and with appropriate tools installed:

docker run -t --pid=container:caddy \
  --net=container:caddy \
  --cap-add sys_admin \
  --cap-add sys_ptrace \
  strace

Assuming you make it so when building your strace container, you'll now have a shell with appropriate tools, from which you can run ps and see the process in the caddy container, and you can run strace against it.

You will be in a different container, with a different file system, but you can see the file-space of the target container at /prof/$PID/root.

Solution 3

I just managed to strace a docker container using these steps:

1. Work out what distro image the container is based on, then obtain the strace binary from that distro, e.g. by installing the corresponding distro package from a container created for that purpose.

2. Copy the strace binary into a volume you can mount into the container.

3. Also create a small wrapper shell script called entry.sh which contains your strace invocation. In my case I wrote it like this:

   #!/bin/sh
   exec /path/to/strace -ff -o /path/to/dumps /bin/bash /original/entrypoint
   

   This is assuming that the original entry point, which you read from the Dockerfile of the image you want to debug, started with #!/bin/bash. Make sure you set the execute bit of this script, and place it where you also placed the strace binary.

4. Launch docker using a command like this:

   docker run -v $PWD/shared:/path/to \
              --entrypoint="/path/to/entry.sh" \
              --cap-add SYS_PTRACE \
              image-name
   

The mounted volume will make the strace and entry.sh available inside the docker. The entry point will do the strace invocation before calling the actual entry point. This might potentially cause some trouble with strace itself becoming pid 1 in the container, and failing to reap children. If that's a problem, a different approach like the one Phil suggested would be better. Finally that added capability tells docker that it's OK to start tracing processes. Otherwise you'd get error messages like

strace: …: PTRACE_TRACEME doesn't work: Operation not permitted

Actually pointing out this capability setting is the reason I'm writing my answer. I had already done the steps except for this flag, and while searching for a solution there I found both this question here and a blog post by John Goulah containing that bit of information. For the sake of completeness I think the flag should be mentioned here, too. Haven't tried Phil's approach yet, so I definitely don't claim my approach to be superior to what he suggested. I guess it might work more easily on systems where you don't want to mess with the docker daemon.

Solution 4

we can add param --security-opt=seccomp:unconfined

I have try it ,it work well!!

docker run -it   --security-opt=seccomp:unconfined  centos:7 /bin/bash

yum install strace
strace ls

execve("/usr/bin/ls", ["ls"], [/* 8 vars */]) = 0
brk(NULL)                               = 0x1d0a000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0x7ffb588da000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or 
directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3

refer:http://johntellsall.blogspot.com/2016/10/tip-use-strace-to-debug-issues-inside.html

View more solutions

Comments

[Guillaume Delafosse]

Your suggestion implies that strace must be installed in the Docker image I want to monitor, which is not something I could assume.

[Christian Aichinger]

Didn't work for me: unknown flag: --security-opt. Docker version 18.06.1-ce, build e68fc7a.

[Alex]

The docker-prefix has been removed from docker-runc. github.com/docker/for-linux/issues/486 So if somebody is searching for the file to replace, u need to search for runc instead of docker-runc.