FreeBSD Jails or Docker instances

Question

What are the main differences between jails on FreeBSD and Docker on Linux? Is one considerably more secure or more performant than the other? Jails are way older than Docker instances so the code itself could be considered more secure. But Jails never "caught on" so perhaps they're not as good as Docker instances? Or is that just because Linux is so much more popular than FreeBSD?

Solution

I will address some of the aspects of FreeBSD jails and Linux Docker, of how they are similar and how they are different.

• both serve the same goal: it's an implementation of lightweight virtualization, when you run application in a separated and isolated compartment under same kernel, and here the similarities end, and differences begin
• Docker is a tool that is used mainly for running prepared binary images that one can get from various public or private repositories. Most people use it in this way. Fewer people construct their own ones and upload then into those repositories.
• FreeBSD jails are more like LXC in Linux: it's a method of creating your container from scratch. You create it, install the software into it, and that's pretty much all - it can be treated as a FreeBSD inside FreeBSD. There's no easy way to export that container as a single piece of software, like in Docker. So pretty much all the people that use FreeBSD, au contraire, always build their own piece of FreeBSD inside a container from scratch (i.e. from the base system installed), installing software from FreeBSD Ports system or from source. FreeBSD jails thus are less friendly, and have higher starting use cost.
• in the same way as the LXC containers are persistent, FreeBSD jails also are persistent. All the changes remain intact between the boots. As I said, it's just a piece of OS inside the same OS (though userland software versions may differ until the ABI is maintained in the FreeBSD kernel). This means you cannot commit the changes - they are commited once they are created.
• from all of the above it's obvious why FreeBSD containers cannot have their similar orchestration layer, like Docker does: there's no ports publishing, there's no volumes, there's no links or any meta-connections between different jails. Only the interfaces the networked OS gives you: network sockets, unix sockets, common mount points.
• there's no limitations, of course, about who can communicate with whom inside FreeBSD jails, as there are none in Docker.
• you can run native docker images in FreeBSD, because there's at least partial support of docker under FreeBSD, but, since Linux is free, I strongly advise against it (and I truly and deeply love FreeBSD), because they will be ran through Linux compatibility layer (provided with a FreeBSD kernel module) which provides some known limitations.

Comments

[Vrakfall]

For more information about docker on FreeBSD. It used to work within a jail (imagine the double isolation :) ), it was made possible as a port by the FreeNAS' main dev as they wanted to use it on that distro. Here's one article from them talking about it and the old wiki page for it. Since then, they stopped maintaining it and it seems to not work anymore. More to follow in the next comment.

[Vrakfall]

Here is the currently last forum post talking about Docker's state in FreeBSD. Props to Phishfry there for some of the info I dropped here. It's still possible tho to use docker on FreeBSD by running it in a virtualized Linux (bhyve, virtualbox, etc...) and using the client-only port, which is still maintained and working, to connect to it. After all boot2docker was originally made to use docker that way on MacOS.

[Vrakfall]

And we know MacOS is a (shameful, cough, this is my opinionated bit) port of BSD. But all of that, of course, is only for a development environment and can in no way be used in production. This is for those who want to keep working on FreeBSD and be able to develop docker things for their job/hobby/whatever.

[Vrakfall]

Oh and I forgot to say, but you mentioned it, the old docker-freebsd port was indeed using FreeBSD's Linuxulator.

[T. Giri]

It is not just that I want to run public images. For example if I don't want to spend countless hours to learn security I go up to dockerhub and write in "hardened": hub.docker.com/search?q=hardened&type=image and run something from there. It vastly increases productivity compared to building my own FreeBSD container from scratch without being a security expert. But if you have an alternative similarly fast solution, then please share! :-)

[foudfou]

Just to add to the very good answer: - for me the main difference is application-based (starting a single process) vs system-based (managing a whole system/OS). - re: exporting and committing jails, since most people use them with ZFS, it's actually possible, although maybe less straight-forward