What are the benefits of running a docker container inside a VM vs running docker containers on bare metal?

Question

What are the benefits of running a docker container inside a VM vs running docker containers on bare metal (on the host directly)?

I have heard of companies running docker containers inside of a VM, particularly it has been mentioned in docker conferences that some organizations are doing it. Why?

( Comparing Docker container running on host vs Docker container running inside KVM on host )

• Both Docker and KVM have ways to save their current state, no added benefit here
• Both Docker and KVM can be provided separate IP's for network use
• Both Docker and KVM separate running programs and installs from conflicting with host running processes
• Both Docker and KVM provide easy ways to scale with enterprise growth
• Both Provide simple methods of moving instances to different hosts

So why would anyone run Docker inside a KVM? Wouldn't they be taking a unnecessary performance hit from the KVM?

Solution 1

Regarding your main points:

Both Docker and KVM have ways to save their current state, no added benefit here

Except that how they store their state is different, and one method or the other may be more efficient. Also, you can't reliably save 100% of the state of a container.

Both Docker and KVM can be provided separate IP's for network use

Depending on what VM and container system you use, this may be easier to set up for VM's than for containers. This is especially true if you want a dedicated layer 2 interface for the VM/container, which is almost always easier to do with a VM.

Both Docker and KVM separate running programs and installs from conflicting with host running processes

VM's do it better than containers. Containers are still making native system calls to the host OS. That means they can potentially directly exploit any bugs in those system calls. VM's have their own OS, so they're much better isolated.

Both Docker and KVM provide easy ways to scale with enterprise growth

This is about even, though I've personally found that VM's done right scale a bit better than containers done right (most likely because VM's done right offload the permissions issues to the hardware, while containers need software to handle it).

Both Provide simple methods of moving instances to different hosts

No, not exactly. Both can do offline migration, but a lot of container systems can't do live migration (that is, moving a running container from one host to another). Live migration is very important for manageability reasons if you're running at any reasonable scale (Need to run updates on the host? Migrate everything to another system, reboot the host, migrate everything off of the second host to the first, reboot that, rebalance.).

Some extra points:

• VM's generally have easier to work with high-availability options. This isn't to say that containers don't have such options, just that they're typically easier to work with and adapt application code to with VM's.
• VM's are a bit easier to migrate directly to and from cloud hosting (you don't have to care to quite the same degree what the underlying hosting environment is like).
• VM's let you run a different platform from the host OS. Even different Linux distributions have sufficient differences in their kernel configuration that stuff written for one is not completely guaranteed to work on another.
• VM's give you better control of the potential attack surface. With containers, you just can't get rid of the fact that the code for your host OS is still in memory, and therefore a potential attack vector. With VM's, you're running an isolated OS, so you can strip it down to the absolute minimum of what you actually need.
• Running a group of related containers together in a VM gives you an easy foolproof way to start and stop that group of containers together.

Solution 2

Virtualization came to be when hardware became to large for single loads, and a need to make a single powerful machine do more than one task, with the tasks being isolated from each other became apparent. VMs do a great job doing exactly that - splitting a large host into multiple smaller hosts. Containers are a way not to split a host into usable chunks, but a way to isolate specific apps and workloads.

So, having a powerful machine, you might want to split it into several smaller parts with maybe different properties (resources, OS etc), and then run your apps inside those VMs as it makes sense for the use case.

This is not the absolute and only way to do things of course, but it is the most obvious case for mixing VMs and containers

Comments

[Ulrich Schwarz]

Never underestimate structural inertia – "we already have all the workflow and infrastructure for VMs working and in place, but the customer insists they can only do docker, so we give them a VM they can run their docker image in".

[A.B]

Docker security model is a bit more flaky: if you can access the "Docker socket" you can consider yourself as root on the system (eg: by running a privileged container that will mount / and remove the hosts's root password)

[Amitav Pajni]

VMs get even more advantageous when you start doing multiple-machine orchestration with something like Kubernetes.

[Austin Hemmelgarn]

@MichaelHampton It really depends on the orchestration tool and the underlying container or VM technology. Ansible for example is actually more efficient working with local Docker containers for example than VM's, because it has lower connection overhead.