Linux LXC vs FreeBSD jail

Question

Are there any notable differences between LXC (Linux containers) and FreeBSD's jails in terms of security, stability & performance?

On first look, both approaches look very similar.

Solution

No matter the fancy name used here, both are solutions to a specific problem: A better segregation solution than classic Unix chroot. Operating system-level virtualization, containers, zones, or even "chroot with steroids" are names or commercial titles that define the same concept of userspace separation, but with different features.

Chroot was introduced on 18 March 1982, months before the release of 4.2 BSD, as a tool to test its installation and build system, but today it still has its flaws. Since the first objective of chroot was only to provide a newroot path, other aspects of system that needed to be isolated or controlled got uncovered (network, process view, I/O throughput). This is where the first containers (User-level virtualization) appeared.

Both technologies (FreeBSD Jails and LXC) make use of userspace isolation to provide another layer of security. This compartmentalization will ensure that a determined process will communicate only with other processes in the same container on the same host, and if using any network resource to achieve "outside world" communication, all will be forwarded to the assigned interface/channel that this container has.

Features

FreeBSD Jails:

• Considered stable technology, since it is a feature inside FreeBSD since 4.0;
• It takes the best of ZFS filesystem at the point where you could clone jails and create jail templates to easily deploy more jails. Some more ZFS madness;
• Well documented, and evolving;
• Hierarchical Jails allow you to create jails inside a jail (we need to go deeper!). Combine with allow.mount.zfs to achieve more power, and other variables like children.max do define max children jails.
• rctl(8) will handle resource limits of jails (memory, CPU, disk, ...);
• FreeBSD jails handle Linux userspace;
• Network isolation with vnet, allowing each jail to have its own network stack, interfaces, addressing and routing tables;
• nullfs to help linking folders to ones that are located on the real server to inside a jail;
• ezjail utility to help mass deployments and management of jails;
• Lots of kernel tunables (sysctl). security.jail.allow.* parameters will limit the actions of the root user of that jail.
• Maybe, FreeBSD jails will extend some of the VPS project features like live migration in a near future.
• There is some effort of ZFS and Docker integration running. Still experimental.
• FreeBSD 12 supports bhyve inside a jail and pf inside a jail, creating further isolation to those tools
• Lots of interesting tools were developed during the last years. Some of them are indexed on this blog post.
• Alternatives: FreeBSD VPS project

Linux Containers (LXC):

• New "in kernel" technology but being endorsed by big ones(specially Canonical);
• Unprivileged containers starting from LXC 1.0, makes a big step into security inside containers;
• UID and GID mapping inside containers;
• Kernel namespaces, to make separation of IPC, mount, pid, network and users. These namespaces can be handled in a detached way, where a process that uses a different network namespace will not necessarily be isolated on other aspects like storage;
• Control Groups (cgroups) to manage resources and grouping them. CGManager is the guy to achieve that.
• Apparmor/SELinux profiles and Kernel capabilities for better enforcing Kernel features accessible by containers. Seccomp is also available on lxc containers to filter system calls. Other security aspects here.
• Live migration functionality being developed. It’s really hard to say when it will be ready for production use, since docker/lxc will have to deal with userspace process pause, snapshot, migrate and consolidate - ref1, ref2. Live migration is working with basic containers(no device passthrough neither complex network services or special storage configurations).
• APIs bindings to enable development in python3 and 2, lua, Go, Ruby and Haskell
• Centralized "What's new" area. Pretty useful whenever you need to check if some bug was fixed or a new feature got committed. Here.
• An interesting alternative could be lxd, that under the hood works with lxc but, it has some nice features like a REST api, OpenStack integration, etc.
• Another interesting thing is that Ubuntu seems to be shipping zfs as the default filesystem for containers on 16.04. To keep projects aligned, lxd launched it's 2.0 version, and some of the features are zfs related.
• Alternatives: OpenVZ, Docker
• Docker. Note here that Docker uses namespaces, cgroups creating "per app"/"per software" isolation. Key differences here. While LXC creates containers with multiple processes, Docker reduces a container as much as possible to a single process and then manage that through Docker.
• Effort on integrating Docker with SELinux and reducing capabilities inside a container to make it more secure - Docker and SELinux, Dan Walsh
• What is the difference between Docker, LXD, and LXC

Docker no longer uses lxc. They now have a specific lib called runc that handles the integration with low-level Kernel namespace and cgroups features directly.

Neither technology is a security panacea, but both are pretty good ways to isolate an environment that doesn’t require Full Virtualization due to mixed operating systems infrastructure. Security will come after a lot of documentation reading and implementation of kernel tunables, MAC and isolations that those OS-Level virt offer to you.

See Also:

• Hand-crafted containers
• BSD Now: Everything you need to know about Jails
• ezjail – Jail administration framework
• A Brief History of Containers: From the 1970s to 2017
• Docker Considered Harmful - Good article about the security circus around container technologies.

Comments

[Pavel Šimerda]

LXC is a rather new technology, so I would expect better security and stability with jails. Not even a guess about performance. There are some known security issues with LXC that can be mitigated using selinux, for example. I personally like LXC, though.

[Russ Jackson]

@PavelŠimerda I just heard of LXC today, but I was suprised to find outh that both Heroku and probably Google App Engine already use LXC.

[user134706]

If you've just bumped into LXC you should take a look at Docker which uses LXC under the bonnet: docker.io/the_whole_story

[0xC0000022L]

@nwildner it doesn't use liblxc anymore, but it uses the exact same concepts: kernel namespaces.

[allquixotic]

Worth mentioning that out of the box, there is no effort by lxc to provide proper guest isolation. lxd, however, does attempt to provide isolation like BSD jails or Solaris zones.

[Admin]

lxd is a "better experience" of LXC, putting other features on top of it. However, it seems nice to quote this little guy in here

[0xC0000022L]

@allquixotic you mean if using unchanged configuration created from templates? True, but unprivileged (userns-enabled) containers were available with LXC 1.x and could even be auto-started at system boot, provided that they were owned by root (and thus located in the system-wide location for containers).