What does this docker log entry mean?

Question

Some minutes ago my computer crashed because apparently my hard drive was full. After booting in recovery mode I then found out that my syslog file in /var/log was 64GB large. I saved the end of the file on another partition and then deleted it. Apparently docker seems to have been the problem because I found a lot of this at the end of the file and there was a docker process running at 200% CPU the whole time. After clearing the log and killing docker everything seems to be normal.

Nov 15 01:44:08 Elemental docker.dockerd[1120]: 
time="2019-11-15T01:44:08.727060251Z" level=error 
msg="failed to get event" error="rpc error: code = 
Unavailable desc = all SubConns are in TransientFailure, latest connection 
error: connection error: desc = \"transport: Error 
while dialing dial unix /run/containerd/containerd.sock: 
connect: permission denied\"" module=libcontainerd namespace=plugins.moby
Nov 15 01:44:08 Elemental docker.dockerd[1120]:     time="2019-11-15T01:44:08.727116701Z" 
...

And so on. I hope this problem won't appear again but still I would like to know what might have happened here.

Solution 1

I had docker package installed through apt and snap at the same time. So the issue is fixed by removing docker through the snap packaging system.

# apt list --installed | grep docker   
docker/bionic,now 1.5-1build1 amd64 [installed]
docker-ce/bionic,now 5:19.03.5~3-0~ubuntu-bionic amd64 [installed]
docker-ce-cli/bionic,now 5:19.03.5~3-0~ubuntu-bionic amd64 [installed,automatic]

# snap list | grep docker
docker                18.09.9                     418   stable    canonical✓  -

# snap remove docker
docker removed

Solution 2

Someone at Cannonical screw things up few hours ago - it's tracked on launchpad - https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1852720

My system was installed yesterday with docker only and same as rest of folks here - syslog killed entire free space on disk, purging docker from snap helped.

Solution 3

You have multiple docker services running. Looks like dockerd is trying to access the different installation of containerd, spamming syslog.

In my case, the teamcity agent and docker installation was conflicted.

sudo apt-get purge docker-ce
sudo apt autoremove
sudo rm -rf /var/lib/docker
sudo truncate -s 0 /var/log/syslog

Ran those commands above and restarted. everything went okay.

Solution 4

Happened to me too. I wonder if it was a snap updated that made it happen:

ID   Status  Spawn               Ready               Summary
17   Done    today at 00:34 UTC  today at 00:34 UTC  Auto-refresh snap "docker"

Uninstall+reinstall of Ubuntu's docker.io package brings back this problem, but for some reason I also had a snap of docker installed, so that's what triggered the conflict.

Name    Version    Rev   Tracking  Publisher   Notes
core    16-2.42.1  8039  stable    canonical✓  core
docker  18.09.9    418   stable    canonical✓  -

However, I don't remember that I ever installed the docker snap package... but this is a test-system after all, someone else might have thought that it was a good idea...

Solution 5

I ran into this problem too, so just in case other people wander in here I can describe the solution that worked for me. I first tried Chris Sung's suggestion but that did not work.

The symptoms were:

1. dockerb running at 200+% CPU, and would constantly restart with a kill command

2. Giant files being filled in syslog as Philip Z. saw. When I came in this morning it had filled up my entire hard drive with a 700GB file.

Frist remove the huge file so you can actually do stuff. It will start filling up again, but you should have some time.

sudo truncate -s 0 /var/log/syslog

Then remove the snap docker installation. This was the problem for me, not docker-ce

sudo snap stop docker
sudo snap remove docker

I'm not sure it's necessary, but I went ahead and got rid of snap altogether too

sudo apt purge snap

You should no longer see dockerb in top. You can then rerun the log truncator to remove any junk that was written while you were doing the above commands. If you screw up as I did and delete the syslog altogether, make sure that you give the new syslog the correct permissions.

sudo cd /var/log
sudo touch syslog
sudo chown syslog:adm syslog
sudo service rsyslog restart

View more solutions

Comments

[ahmadalibaloch]

I was getting disk full warnings and system was not logging in due to huge syslog file. I deleted the docker from snap to solve it.

[Duncanmoo]

Thank you, my machine was freaking out this morning as a result of this.

[Belly Buster]

Likewise, I didn't realise I had docker installed via snap and had issues starting at 04:00Z today. I needed to run sudo snap remove docker --purge to fix it. Thanks for the response.

[Try431]

Odd that this happened to me at just about the same time as you! I wonder if some update got pushed - in any case, sudo snap remove docker --purge fixed my high CPU issues!

[Jaymes Bearden]

What happened that allowed Cannonical the ability to remotely install packages. This is actually ridiculous and brought down quite a few of my own servers

[Chris Heald]

This hit me as well. docker auto-installed itself via snap earlier today, which generated 800GB of logs and filled up our root disk. WTF?