Where to install ESXi so that all local drives are available to VSAN

Question

I am currently setting up a new 3-host VSAN cluster that will be placed into production soon. (Note this is the newer "Virtual SAN" (VSAN) technology, not the older vSphere Storage Appliance (VSA) technology). This is the first time I have worked with VSAN.

Each of the three hosts in the cluster has four 1TB local HDD's and one 200GB local SSD (which VSAN needs for read/write caching) to contribute to the cluster. I have installed ESXi 5.5 directly on the first local HDD of each host.

I added the three hosts to vCenter and launched the vSphere Web Client to configure VSAN. But instead of seeing all four local HDD's on each host as being available to VSAN, only three are available.

From what I have read in the VSAN documentation, disks used by VSAN must solely be used by VSAN. That is, once VSAN "takes over" a local drive, that drive can not be used for any other purposes (such as reserving partitions for use by other OS's). However, it wasn't clear to me if that also meant not being able to store the ESXi bootable partition on one of the VSAN local drives.

I've heard a lot about installing ESXi to a bootable USB stick as an option, but have never tried it since we've always just placed the ESXi boot partition on our traditional SAN. (For background, the new VSAN cluster is going to replace our blades and traditional SAN due to downsizing and cost-cutting measures).

So to summarize, here are my questions:

1. Is it possible to install ESXi to one of the local HDD's and still have the rest of the space on that HDD be available to VSAN?
2. If the answer to question #1 is "no", then is the next best option to install ESXi to a bootable USB stick? (and if so, are there any notable tradeoffs to doing this that I should be aware of?)

Solution 1

The VSAN design guide recommends the following:

• Use SD, USB, or hard disk devices as the installation media whenever ESXi hosts are configured with as much as 512GB memory.

• Use a separate magnetic disk or solid-state disk as the installation device whenever ESXi hosts are configured with more than 512GB memory.

---

My recommendation for your specific Supermicro configuration is to use the internal USB with an enterprise purpose-built USB card (there should be a slot on the motherboard). If not, and your hosts are using a SAS HBA or motherboard SATA ports, and you have spare drive slots, using a small single SSD is an okay alternative.

In terms of the caveats associated with USB/SDHC card boot in vSphere, my experience is here:
What happens when the USB key or SD card I've installed VMware ESXi on fails?

VSAN accounts for this situation reasonably well.

Solution 2

1. You can, but you don't really want to, because then you have one disk smaller than the rest, and you can't use passthrough if you wanted to.

2. Most servers have an internal SD-card slot (or sometimes mirrored slots), or an internal USB port directly on the motherboard. You can install it on one of these. I have a bunch running off the SD card, and two that use USB stick.

ESXi is smart enough to know that it's installed on flash media. Have a look at this KB from VMWare.

If you do end up on USB instead of SD, make sure you buy a good, proper flash drive that has a nice fast write speed. I just spent my entire day copying 60GB of files onto a 64GB flash drive that could only write at 3MB/sec. It was horrific. The USB sticks that cost twice the price typically get way, way more than twice the performance.

Comments

[Admin]

A lot of servers have an internal SD/Micro-SD/USB port for exactly this reason, to boot ESXi off - we do it and it works just fine.

[Admin]

@ewwhite You are right about the VSAN optimization. We purchased one of the preconfigured "VSAN Ready Nodes" to make sure there were no issues with VSAN compatibility. Specifically, we got the SuperMicro SuperServer 1018D-73MTF Ready Node in a pretty much stock configuration (see page 4 of the PDF at this link and here), with the only change being that we went with 4 1TB HDD per host instead of 5.

[MrGigu]

I wouldn't recommend this for vsan deplouments. If every host powers down because of say, a power outage, you'll never get any host back online because you won't be able to boot any of the vsan nodes, because they all rely on having at least one other node online.

[Mario Lenz]

When all nodes are down and VSAN nodes require at least one other node online to boot, it shouldn't matter whether I can't boot the hosts from HDD or whether I can't boot them via network. Maybe there's an error in my reasoning and I'm not seeing something obvious.

[RSW]

@MarkHenderson This is troubling if true. Can I ask where you found this information? I never saw anything about this in the VSAN documentation (I haven't read it all though). Certainly some people are going to have unexpected shutdown of the whole VSAN cluster like this...are you saying that there is no road to recovery (other than restoring backups) if that happens?

[MrGigu]

I think there's been some misinterpreting going on here. What I'm saying is that if you choose to boot your ESXi install off the SAN, but your SAN is running exclusively under ESXi, then you will be in trouble. If you use SD cards, or a different SAN to boot your ESXi hosts from, then you'll be fine. But if you're exclusively VSAN then don't boot from the same VSAN you're hosting from.

[RSW]

@MarkHenderson Ahh...got it! Man, was I concerned I had missed a gaping hole for a minute!

[Mario Lenz]

@MarkHenderson I'm pretty sure you can't boot an ESXi host from VSAN anyway; as far as I understand it's exclusively for the VMs. Btw: My answer to the question was about booting from network, so why do you talk about booting from SAN in your comment(s)?

[MrGigu]

Because if you have an exclusively vsan network, where is that boot server going to live? On a vm stored on the vsan, probably.

[Mario Lenz]

@MarkHenderson Now I understand. Yes, if the boot server runs on the cluster you wouldn't be able to boot the hosts once everything is down. It would have to be hosted somewhere else, e.g. on a dedicated management cluster.