Running Vagrant Inside VMWare VM

Question

I realize this is essentially OSCeption (Operating System Inception), but I think it might make the most sense for me (please tell me if there's a better option, this seems really awful).

Here's the situation: I have a windows 8 machine. I like it - it works great for everything but development. For development, I've been using a VMWare virtual machine running Ubuntu. I've dabbled with using Cygwin, but it just didn't feel right.

I'm now joining a project where they've been using Vagrant to manage development environments so I need to be able to use Vagrant. But, from what I've seen, Vagrant is mainly used to run code within a consistent environment, but not necessarily to write it. And if I wanted to write code by SSH'ing into my vagrant boxes, then I would have to re-configure my preferences like my .vimrc file and what not for every machine.

Does it then make sense to install Vagrant within my Ubuntu VirtualMachine? I feel like at some point VMs within VMs will get out of hand and cause problems. Is there any better way to do this?

Edit: So I tried it out - as I expected I hit some errors. When I try and boot the machine, I get the following error message:

Failed to open a session for the virtual machine vagranttest_1371583212.

VT-x is not available. (VERR_VMX_NO_VMX).

Result Code: NS_ERROR_FAILURE (0x80004005)
Component: Console
Interface: IConsole {db7ab4ca-2a3f-4183-9243-c1208da92392}

Looks like my vmware virtual machine can't run another virtual machine. Any ideas on the best way to go about doing this?

Solution 1

I've run into the same issue today. The solution is quite simple.

1. Power off vmware vitrual machine.
2. Go to "edit virtual machine settings"
3. Go to processors. There are three checkboxes there.
4. Check second checkbox (enable VT-x/AMD-V)
5. Power on machine.

After this virtualbox should work inside vmware.

Solution 2

To answer the original question as well as @blong's Vagrant forum post, this is what I've done to make this work.

I was trying to do something similar myself (actually Vagrant/VMware hosting Vagrant/Vbox) and I have performed all the optimizations I can think of, giving my "host" VM a large amount of RAM (24GB) and 6 cores, disabling swapping VMs to disk (this KILLS things on Windows when it happens) by setting "Fit all VM memory into reserved host memory", and allowing per VM page files (otherwise they live in the system page file which limits how many VMs you can run at once).

What I am doing has been working perfectly, the networking issues I've had were due to the corporate proxy I'm behind. Once I configured that my VM got internet access and all was right with the world.

I did have to manually set --natbindip1 and --natnet1 via the Vagrantfile in addition to the natdnsproxy1 and naddnshostresolver1 that were already set in my example (Virtualbox) Vagrantfile. These settings can be found in the Virtualbox documentation for the correct usage.

To sum it up, use the VT-x passthrough/"virtualize" option in your VM CPU settings, give the VM adequate memory, don't allow that memory to be swapped on the "root" host machine, and try to make sure your network ranges don't overlap or you'll have routing trouble.

Here is the Vagrantfile I was working from, it is based almost entirely on andreptb's gist for the modern.ie vagrant setup. https://gist.github.com/andreptb/57e388df5e881937e62a

# -*- mode: ruby -*-
# vi: set ft=ruby :

# box name into env var, same script can be used with different boxes. Defaults to win7-ie11.
box_name = box_name = ENV['box_name'] != nil ? ENV['box_name'].strip : 'win7-ie11'
# box repo into env var, so private repos/cache can be used. Defaults to http://aka.ms
box_repo = ENV['box_repo'] != nil ? ENV['box_repo'].strip : 'http://aka.ms'

Vagrant.configure("2") do |config|
  # If the box is win7-ie11, the convention for the box name is modern.ie/win7-ie11
  config.vm.box = "modern.ie/" + box_name
  # If the box is win7-ie11, the convention for the box url is http://aka.ms/vagrant-win7-ie11
  config.vm.box_url = box_repo + "/vagrant-" + box_name
  # big timeout since windows boot is very slow
  config.vm.boot_timeout = 500

  # rdp forward
  config.vm.network "forwarded_port", guest: 3389, host: 3389, id: "rdp", auto_correct: true

  # winrm config, uses modern.ie default user/password. If other credentials are used must be changed here
  config.vm.communicator = "winrm"
  config.winrm.username = "IEUser"
  config.winrm.password = "Passw0rd!"

  config.vm.provider "virtualbox" do |vb|
    # first setup requires gui to be enabled so scripts can be executed in virtualbox guest screen
    #vb.gui = true
    vb.customize ["modifyvm", :id, "--memory", "1024"]
    vb.customize ["modifyvm", :id, "--vram", "128"]
    vb.customize ["modifyvm", :id,  "--cpus", "2"]
    vb.customize ["modifyvm", :id, "--natdnsproxy1", "on"]
    vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
    vb.customize ["guestproperty", "set", :id, "/VirtualBox/GuestAdd/VBoxService/--timesync-set-threshold", 10000]
  end
end

My additional changes:

# Need the WinRM gem for managing from Linux
$ sudo gem install winrm

    config.vm.communicator = "winrm"
+  config.winrm.host = "localhost"
    config.winrm.username = "IEUser"
    config.winrm.password = "Passw0rd!"
# This one may not be necessary, I added it for completeness
+  config.vm.guest = :windows

# In order to USE the two CPUs you need the ioapic
# Virtualbox gives an error in the GUI and only shows 1 CPU in the VM otherwise
      vb.customize ["modifyvm", :id, "--cpus", "2"]
+    vb.customize ["modifyvm", :id, "--ioapic", "on"]
# We had to modify the network range because we are running Virtualbox inside VMware
+    vb.customize ["modifyvm", :id, "--natnet1", "192.168.199.0/24"]

Remove the + signs and add those lines into the Vagrantfile above and you should have an equivalent working system to what I've been using.

Solution 3

If you are running virualbox in a VM in vsphere, you have to ssh to the ESXi to update a configuration.

Steps:

1. ssh to ESXi server.
2. find the vmx file which belong to your VM find / -name *.vmx
3. poweroff your VM.(very important, or your change will be overwrite)
4. edit that vmx, append a new configuration at the bottom of the file: vhv.enable = "TRUE"
5. power on your VM
6. enjoy the Vagrant. :)

Solution 4

I have tried this in two VMware products. Right-click on the VM:

• In vCloud Director 5.5 VM properties on the Hardware tab there's an "Expose hardware-assisted CPU virtualization to guest OS" checkbox, but it's grayed out for me. YMMV.
• In vSphere Version 5.5.0 Edit Settings > Virtual Hardware > CPU the checkbox is called "Expose hardware assisted virtualization to the guest OS," and that worked for me.

View more solutions

Comments

[Terry Wang]

Looks like you are trying to run embedded type 2 hypervisor. First, not all hypervisors support running another hypervisor (same or diff) inside it. 2nd, have you actually enabled the option in the VM's Settings - CPU - Visualize Intel VT-x or AMD-V?

[Terry Wang]

BTW: VirtualBox inside VirtualBox is possible as long as you enable the VT-x/AMD-V acceleration.

[Matthew Skelton]

I can confirm Terry's comment - at my previous place we ran Vagrant VMs inside vSphere VMs in order to build & test new core Vagrant box images :)

[blong]

@MatthewSkelton Have you tried VirtualBox inside a VMWare Fusion VM? Are there any gotchas? I'm having a heck of a time trying to configure a CentOS host running in VMWare to run Vagrant/VirtualBox quickly inside of it, posted to the mailing list here: groups.google.com/d/msg/vagrant-up/6YHdRupCKuI/2h2B4encffQJ

[blong]

Same question @TerryWang

[Matthew Skelton]

@b.long Not tried running VirtualBox inside VMware Fusion. It would surprise me if the support for VMware Fusion lags behind the support from vSphere - the virtualization drivers may well be less advanced. Why would you want to double-virtualise with a desktop hypervisor?

[blong]

@MatthewSkelton Thanks for getting back to me. The reason for double-virtualization at this point is to build out the CentOS host that can then be exported and moved to a vShpere environment. Once I'm sure the CentOS host is configured properly, it'll be moved to the server hypervisor and no longer be run it in VMWare Fusion. I'm using VMWare Fusion only to configure and check the CentOS host before exporting it.

[Matthew Skelton]

@b.long - what we did is simply to convert from the VirtualBox file format to the vSphere file format before spinning up in vSphere and only then run acceptance tests. That meant that as soon as the acceptance tests passed we could promote the candidate VM image to a template immediately. Details here: owainperry.com/2013/09/04/…

[Jörg Brenninkmeyer]

@TerryWang I have a 64 Bit Ubuntu with VT-x/AMD-V acceleration enabled running under Windows 8.1. When I install the latest Virtualbox 4.3.14 inside the Ubuntu VM, it offers me only 32-bit machines and VT-x/AMD-V acceleration is disabled. This in turn doesn't allow me to use the vagrant machine inside Ubuntu. How did you make VirtualBox inside VirtualBox work with VT-x/AMD-V acceleration enabled?

[Ytsen de Boer]

@Terry Wang: VirtualBox inside VirtualBox is not implemented. Source: virtualbox.org/ticket/4032. It's good to hear that VirtualBox can run inside VMWare at least, with the settings provided below by tbc0.

[Jasmine Hegman]

Just ran across this and wanted to update this comment thread re: virtualbox.org/ticket/4032. It is now implemented since VirtualBox 6.1.0

[Andrew Orsich]

@b.long: No, it not sounded rude actually. I hope that my solution will help you :)

[blong]

Have you had any issues doing this with CentOS installed on the VMWare VM and using VirtualBox inside CentOS? My inner VM runs slow, no matter what I do. I don't expect this, since I gave the CentOS VM 7140MB of memory, and 2 cores from my 2.7GHz Core i7. When I try to startup my inner VM's (with or without Vagrant) they run slowly, and don't seem to allocate much memory. As the VM is booting, I see CPU 1 & 2 spike a bit, then eventually drop, but the memory usage is never above 1GB. When trying to start the VM, it times out like this: gist.github.com/b-long/3ef3a726f9f4d28f4d59

[blong]

I've posted my issue to Vagrant's mailing list as well: groups.google.com/forum/#!topic/vagrant-up/6YHdRupCKuI

[Erik]

Just a note for anyone with ESXi's free version.. this is not possible unless you upgrade your virtual machine's hardware to v9+ .. and once that occurs you're unable to edit VM details from the vSphere Client.

[ET Worker]

Thanks for the simple solution, it works like a charm

[v0d1ch]

I had to add v.customize ["modifyvm", :id, "--cpus", "2"] v.customize ["modifyvm", :id, "--hwvirtex", "on"] . @dragon788 How did you set IP address ?

[dragon788]

@vodich I did so much the same as the other parameters. ["modifyvm", :id , "--natbindip1", " true"]. I'll double check the syntax, but basically it takes the modify command, the VM ID, the parameter to change, and the value to set it to.

[v0d1ch]

Thanks, I think I am going to ask a quetion in a new thread about configuration other than what we discussed here since I just spent 6 hours trying to figure it out without success :(

[v0d1ch]

Can you paste the other settings from your vagrantfile so I try with that ?

[Abhishek Dujari]

This is the best solution for all VMWare products. I am using Workstation 11 and I had not reverted my virtualization setting after a system upgrade test. It was manually set to binary translation.

[dh762]

If it still doesn't work, try to reduce the vm.cpus to 1: stackoverflow.com/a/24776690/1344855

[Iman Kermani]

worked for me with ESXi client 5.5 and Ubuntu 16.04 as guest machine