Save docker-compose logs to a file

Question

I have unit tests running on my build server and would like to capture the log results for analysis when something fails. I have yet to find a way to redirect the output of docker-compose logs to a file, or to find where the log files themselves actually live.

I want the equivalent of:

docker-compose logs > logs.txt

---

Edit - clarification:

All of my docker containers produce useful logs, which a manual run of docker-compose logs reveals. I want to script this process to save those same logs to a file that is an artifact on my build server. Essentially, the output of docker-compose logs saved to a file, however docker-compose logs never exits.

Solution 1

By default docker uses the json-file driver to record your containers logs and the raw json output of the logs can be found in:

/var/lib/docker/containers/[container-id]/[container-id]-json.log

You can get this location by running:

docker inspect --format='{{.LogPath}}' [container-id or container-name]

When you run docker-compose logs [service-name], docker-compose will attach to the service (container) you reference and the LogPrinter object will output the contents of the above file, but formatted so they're easier to read.

Related docs: https://docs.docker.com/compose/compose-file/#logging

Solution 2

I am not sure what you are trying to achieve. Are you trying to reproduce

docker-compose logs > logs.txt

within the compose file as an instruction? Or is your issue that the redirect does not "catch" the whole output?

In the later, you can do:

docker-compose logs --no-color >& logs.txt

Or

docker-compose logs --no-color |& tee logs.txt

to both see the logs on the terminal and dump it to a file at the same time.

Solution 3

In later release docker-compose 1.7.x+, it is fixed. see https://github.com/docker/compose/issues/2227 & https://github.com/docker/compose/releases/tag/1.7.0

Before that, there is another way to achieve it, the solution of accepted answer is to access host files directly, Which may be not applicable for remote/security case.

Below we can get the container name from docker-compose ps command and let the docker logs command to loop

docker-compose ps | tail -n +3 | awk '{print $1}' | xargs -n1 docker logs

Solution 4

The docker-compose logs command does terminate (unless you add the --follow setting).

The likely problem here is that the logs are so large it takes some time to format and output them. You can reduce this issue if you limit the output by specifying either/both the number of lines you want to read and the container you want to read from.

Try:

docker-compose logs --no-color --tail=1000 <service-name> > logs.txt

NOTE: service-name is name of service taken from docker-compose file - not container name or container id

(Adjust the tail number as required. I've added "--no-color" to simplify the output but it is not needed.)

Solution 5

The solution I made is based on Chris McKinnel's answer with some enhancements.

I needed to dump last 1MIO records from my nginx container and docker-compose logs --tail 1000000 nginx > last1mio.log was too slow and the output was not exactly the same as usual nginx files have.

This is solution that works for me:

docker inspect --format='{{.LogPath}}' nginx-1 \
| xargs tail -n 1000000 \
| jq -r -j .log > last1mio.log 2>&1

Notes:

• you need to know your container name, in my case is nginx-1 - I use docker-compose option container_name: for having this constant
• I am using JQ utility to parse json - with some special options to get the format the same as original nginx log files usually have.

In my case dump of last 1mio lines of nginx log takes cca 10s.

View more solutions

Comments

[Ryan]

Typing docker-compose logs starts streaming the log output, right? I basically want to dump all that output to a file. The problem with the command I proposed is that docker-compose logs doesn't terminate, so it doesn't ever redirect output to the file.

[creack]

the logs are a stream, the redirect populates the file as it receives data. Your file sticks with 0 bytes? Does it do that as well with >& instead of > ?

[Ryan]

Your suggestions do work (they populate logs.txt with my logs), but the command never returns, as docker-compose logs runs indefinitely.

[Ryan]

If you run docker-compose logs, does the command ever exit on its own?

[creack]

I am not familiar with compose, the docs are very bad for this particular command (no behavior explanation). I think if you run docker-compose logs <servicename> then it should terminate when the container stops.

[Ryan]

I suppose I could do something a bit gross, like: docker-compose logs >& log.txt & (note the & to background it), then sleep 5; docker-compose stop; but I was hoping for something a little less sloppy.

[Serguei Fedorov]

docker-compose logs --no-color >& logs.txt blocks input on MacOS (possibly linux as well). Doing docker-compose logs --no-color >& logs.txt & will free bash to continue executing other commands.

[Matthew Wilcoxson]

Update: This was fixed in Docker-compose 1.7. (see @larry-cai answer)

[Redsandro]

This is a clever answer, but it does require su privileges by default on Ubuntu because the log directory is only readable by root.

[Joe Huang]

I can't see any *-json.log at there, should I stop the container first?

[Kaszanas]

It seems like the second command is not working as intended as it is resulting in a following error message on Windows machine: Template parsing error: template: :1:2: executing "" at <.LogPath>: map has no entry for key "LogPath"