How to place Docker images ontop of an NFS share in CoreOS?

Question

I have installed CoreOS which by default using loop-back device with size of 100GB and I have reached that limit very easily.

My first idea was to place all the images on top of my NAS using NFS with simply mounting it on top of the /var/lib/docker directory without any success.

Now I have found this article and its looks like docker creates block device in any way, on top of any kind of storage in order to use it's CoW snapshots and that's probably why I wasn't successful on my first shot with simply mounting NFS share on top of image directory. It kind of freaks me out. Why not use pure NFS with native snapshots of the storage? I hope there is good explanation of doing this. In any way I need to do this freak thing. I'm new to docker and CoreOS and do not know how to do this properly. So the question is: How to mount an NFS share and use it for all the images on CoreOS?

Solution

Good news, you're not the first person with a need to do this. Docker cannot use direct NFS because it needs both support for extended attributes (xttrs) to be able to store options like kernel capabilities, SELinux context, etc. By faking things out by using device mapper to create a virtualized block device on remote storage you are providing the required kernel mechanisms to utilize Docker via a secondary mechanism. While this can be configured via the dm.loopdatasize option, as they explicitly point out on the Docker page "Note: This option configures devicemapper loopback, which should not be used in production.".

In the end this comes down to the requirements on the underlying storage. Docker is merely using mechanisms provided by the kernel and not doing magic on it's own.

Comments

[Ponte]

any success in this quest? I'm trying to do the same as I've got a NAS server and would like too save and manage the storage from there

[BBK]

Hi Brian. Thanks for the reply! So at the bottom line, what production-ready options do we have for storage? I suppose it is only block-storage like FCoE/FC/iSCSI/Dirrect attached Storage? And no NAS available for image sharing between CoreOS cluster nodes?