Running many virtual machines on a single host

Question

I have a need to run a relatively large number of virtual machines on a relatively small number of physical hosts. Each virtual machine isn't doing to much - each only needs to run essentially one basic network service - think SMTP or the like. Furthermore, the load on each is going to be extremely light.

Unfortunately, the numbers are something like 100 virtual machines on 5 physical hosts. Each host is decent enough - core 2 with 2 gigs of ram and a 1tb disk. However, I know just taking a vmware image of ubuntu and throwing on that machine wont get me anywhere near 100 instances and would be something closer to 20.

So, is there any hope for this ratio of images to hosts? Also, which implementation of virtual machine would be best suited for this purpose - ie has efficient overall usage of resources? We mostly use vmware here, but if there is a significant performance advantage that could be gained by switching to Xen or the like, I am sure we would consider it.

Thank you in advance for your insights :)

Note: We ended up using OpenVZ and it worked rather well. The default parameters for an ubuntu template let us run about 40 instances per machine.

Solution 1

there are three main fronts to make those fit:

1. lower overhead. OpenVZ, Vserver, chroot, would be ideal if applicable. if you really need each instance be a real VM with it's own kernel, try KVM/Xen instead of VMWare. may be less mature, but you'll have a lot more flexibility.

2. smaller guests. try Ubuntu JeOS, or roll your own with busybox

3. share as much as possible between guests. try sharing a single R/O image with all the OS, and mount a small R/W image for each guest on /var, /home, /etc, etc

Solution 2

A couple of problems with that...

1. For Vmware server you really need Server hardware unless it's only for testing.
2. Go with a virtualization solution that is bare level like Xen Server, or VMware ESX or ESXi (free) or Hyper-V which isn't bare-level but closer in performance.
3. For 20-1 you will need more RAM. The math doesn't add up. Minimal functional machines need 512 unless it's a perfectly stripped linux that should have at least 256. 20x256= 5gb + 5-10% overhead. Not really going to happen on those specs.
4. For 20-1 you will need more processor. Each machine will have a vCPU. shared on a core 2 means that 10-1 per processor. not good. We run almost 20 on a dual quad core Dell 1950, 16gb RAM. Works great.
5. Whatever you choose, you are going to be oversubscribing memory. Not exactly sure which ones let you. Vmware will, but shows warnings.
6. I've heard but have no proof that XenServer will offer performance benefits, but nobody claims more than 10-20%.

Good luck

Solution 3

Do you really need 100 full-functional operating systems?

Why not take approach web servers use already? I mean virtual web servers/hosts.

For example take Apache HTTPD installed on single physical server hosting many virtual servers using single config file. Plus you'll need DNS configured and/or many virtual network interfaces (eth0:0, eth0:1, ... ,eth0:n) with different IP addresses.

This should work if you really need only several services exposed to the world and the load is not high.

Solution 4

Another possibility is to use a lightweight Linux distribution that can run in very small amounts of memory. Something like DamnSmallLinux or a variation on DDWRT. They can run in as little as 16MB of memory, allowing you to run 20 or more on a single machine.

Solution 5

i've got one quadcore machine running a full desktop and 9 virtual machines. since this is a testing machine i use all sorts of guests. the best on ram usage seem to be debian-kfreebsd, and tiny core linux. tiny core linux uses 10M of ram doing nothing. add a couple of services and it might be 32M, so i could run 32 vm's within 1GB of ram! you have 2GB so lets say you could run 48 machines including a hypervisor and overhead(i'm using kvm.) so with 5 machines we'd be up to 240 machines :D

i think i'm going to try that in a moment :D

btw. you said the vm's whould have a light load, so i didn't count on cpuload or diskload. and those figures have exactly 0 redundancy.

View more solutions

Comments

[Voltaire]

I believe we can go with any Linux derivative. Except gentoo. I want these machines to be setup before the end of the century ;)

[Voltaire]

Actually - im not sure. We might have some os specific requirements for each image that prevent us from doing something like running a bunch of FreeBSD jails.

[Voltaire]

its more like 4-5 images per host, for a total of 20 images.

[Kevin Little]

Well, if you could configure a single Linux kernel that supported all your different apps, then Virtuozo and OpenVZ could pack a lot more VMs into a single physical host. Good luck!

[Voltaire]

You know, that might not be a bad idea. If I understand correctly, for each process, I can assign a virtual interface with a unique IP and if I want process to have their own IP, I configure them to use one of those virtual interfaces.

[Voltaire]

Unfortunately though, I think we actually have a reason to have a full blown OS. However, we might not need 100 of them, perhaps only 40, and then we can use this trick for the rest. Its certainly something to contemplate.

[Rune Jeppesen]

You could also try something with chroot environments if you need further compartmentalization. You'd create little mini filesystems for each isolated process, then launch the process in a chroot jail.

[Ajeeb.K.P]

Good answer, as of year 2014. I say Docker and linux container to be added to your point 1.

[nullException]

@Ajeeb.K.P true, Linux containers (and BSD/solaris variants) are just as low-overhead as chroot; but Docker adds a filesystem overlay which I don't know how much does it affect disk IO.

[Ajeeb.K.P]

I just said it's the new trends. It is more developer friendly. And it reduces the difficulties in deploying an app. Big companies like Google,Canonical(Ubuntu company),Redhat etc. are considering it. I do understand it may be affect IO, but the question was about virtual machine. Docket is a linux container based (at least some concept and code), so i belive it is too an answer. Especially, Redhat is going with Project Atomic and Canonical with Ubuntu Snappy, and i heard google is also interested in it.