cannot start docker daemon (graphdriver issue)

Question

running Ubuntu 16.04.5 LTS 4.4.0-108-generic on a virtual private server My aim is to deploy a meteor application with mup.js, but it fails because of dockerd not being launched. The problem is, i cannot get dockerd to launch after a system reboot.

I tried changing graphdriver as suggested in other threads (Not able to start docker on Ubuntu 16.04.2 LTS (error initializing graphdriver)), switching to aufs or overlay2, but to no avail. I also updated my kernel, purged docker repos, reinstalled docker on my machine.

i have close to no experience working with docker, and the website i'm trying to put back online is part of a show, the last night of which is tomorrow! i must say i'm getting a bit desperate, any help is welcome.

thank you!

docker & dockerd are both version 18.06.1-ce, build e68fc7a

$ sudo dockerd
INFO[0000] libcontainerd: new containerd process, pid: 3488 
WARN[0000] containerd: low RLIMIT_NOFILE changing to max  current=1024 max=1048576
WARN[0000] failed to rename /var/lib/docker/tmp for background deletion: %!s(<nil>). Deleting synchronously 
Error starting daemon: error initializing graphdriver: driver not supported

journalctl -xe yields :

Oct 03 01:22:19 vps332343 systemd[1]: Listening on Docker Socket for the API.
-- Subject: Unit docker.socket has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker.socket has finished starting up.
-- 
-- The start-up result is done.
Oct 03 01:22:19 vps332343 systemd[1]: docker.service: Start request repeated too quickly.
Oct 03 01:22:19 vps332343 systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker.service has failed.
-- 
-- The result is failed.
Oct 03 01:22:19 vps332343 systemd[1]: docker.socket: Unit entered failed state.
Oct 03 01:22:37 vps332343 sudo[3651]:  eboutin : TTY=pts/0 ; PWD=/etc/nginx/sites-available ; USER=root ; COMMAND=/bin/journalctl -xe
Oct 03 01:22:37 vps332343 sudo[3651]: pam_unix(sudo:session): session opened for user root by eboutin(uid=0)

df -tH yields :

Filesystem     Type      Size  Used Avail Use% Mounted on
udev           devtmpfs  970M     0  970M   0% /dev
tmpfs          tmpfs     196M  5.6M  190M   3% /run
/dev/vda1      ext4      9.7G  4.6G  5.1G  48% /
copymods       tmpfs     977M   28K  977M   1% /lib/modules
tmpfs          tmpfs     977M   68K  977M   1% /dev/shm
tmpfs          tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs          tmpfs     977M     0  977M   0% /sys/fs/cgroup
tmpfs          tmpfs     196M     0  196M   0% /run/user/1002
tmpfs          tmpfs     196M     0  196M   0% /run/user/1001

/etc/docker/daemon.json contents :

{"storage-driver":"devicemapper"}

(no other modified config file)

Solution 1

Try to configure devicemapper as a storage driver and clean /var/lib/docker/ folder before docker start rm -rf /var/lib/docker/* (it will delete all your previous containers/volumes/...).

Check any warnings from docker info, when docker will be running - they may help you with additional configuration.

Solution 2

This can also be due to a recent kernel update that might have messed up the graphdriver: devicemapper.

So when rm -rf /var/lib/docker/* and reinstalling Docker does not work. Try reinstalling kernel image and reboot.

$ sudo apt-get install --reinstall linux-image-`uname -r`
$ sudo reboot

Comments

[BMitch]

What is your kernel version? How have you installed docker and what is your dockerd version?

[BMitch]

Also, please include you /etc/docker/daemon.json along with any other modified config files.

[Samuel Hackwill]

well hum hehe we are currently running on kernel 4.4.0-108-generic after downgrading from 4.4.0-137 (we edited the grub file and purged the more recent kernels..). I edited my post with the rest of the info.

[BMitch]

What happens if you remove your daemon.json, or switch to overlay2?

[Samuel Hackwill]

deleting the json file and trying to launch deamond throws the same graphdriver error. (ERRO[2018-10-03T16:11:41.088732386+02:00] [graphdriver] prior storage driver devicemapper failed: devicemapper: Error running deviceCreate (CreatePool) dm_task_run failed Error starting daemon: error initializing graphdriver: devicemapper: Error running deviceCreate (CreatePool) dm_task_run failed)

[Samuel Hackwill]

making a new json file with {"storage-driver":"overlay2"} also throws a graphdriver error : ERRO[2018-10-03T16:08:04.961130302+02:00] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded. storage-driver=overlay2

[BMitch]

Strange, devicemapper shouldn't be the default on Ubuntu. Could be seeing some files in /var/lib/docker that is forcing it to go back to that. Also looking like something isn't right with the kernel, perhaps a version that has the needed modules disabled.

[Samuel Hackwill]

i now have the following error : Error starting daemon: error initializing graphdriver: devicemapper: Error running deviceCreate (CreatePool) dm_task_run failed

[Jan Garaj]

Do you have any free space?

[Light.G]

Aufs is default option in Ubuntu, and it works well. Wondering why you suggesting so?

[Samuel Hackwill]

thank you for your answers. I have root, hum i have to say that i don't know what is my fs (file system?). I have free space but not a massive amount (maybe 5G) as it's a small droplet. regarding aufs, i can't manage to get it installed either, nor does dockerd when i specify it as a graphdriver..

[Jan Garaj]

devicemapper is the most troublefree storage driver. But this looks like an OS issue - maybe you have a custom kernel without full docker support, maybe /var/lib/docker is not mounted with correct parameters, .... 'df -Th' command can be used to check fs type/size.

[Samuel Hackwill]

i reinstalled docker already, so var/lib/docker should be clean? i'm going to look into maybe upgrading the kernel, also i edited post to show the results of df -Th.

[BMitch]

Devicemapper is supported for rhel and centos, and is being phased out because of all the issues and limitations it has. This shouldn't be used on a Ubuntu install.

[Jan Garaj]

This seems to an issue not related to used storage driver. Anyway, yes devicemapper has own issues, but other storage drivers have as well. It is officially supported on Ubuntu as well - docs.docker.com/storage/storagedriver/select-storage-driver/‌​…

[BMitch]

It can be supported and phased out at the same time. Everything is defaulting to overlay2 with a new enough kernel now.