howto plan RAID for ESX

Question

eight 300GB SAS drives are available. Can ESX be put on one disk as RAID-0 and others as RAID-5 ? so that in the event of disk failure data (VMs) are safe. if os disk RAID-0 fails could that be installed on replacement disk and still be able to keep VMs running?

if not RAID-1 for OS is only option for OS disk? please suggest any other RAID options.

All: many thanks for your responses. the question that remains is which RAID? my situation:

1. at max handle 1 disk failure
2. be more storage efficient (less disk cost)
3. better I/O write perf
4. if esxi 4 can handle locking better (Chopper) then why not RAID-6 or 5 for all 8 disks instead of separating OS and data volumes? does this provide ability replace failed disk and continue since OS is now mixed up with data?
5. How good is RAID-50? is it better than RAID-5?

Solution 1

What's wrong with just R10'ing all 8 drives into a single logical disk (LD) of ~1.2TB, it'll be faster and a lot more resilient than your R0+R5 suggestion and gives you the same available space as Tom's R1+R6 suggestion (sorry Tom, you know I love you right). There's no need to have the OS on a different disk than the VMs at all, especially if you're using v4 as it handles locking much better.

EDIT FOLLOWING QUESTION EDIT

Basically you can't have it all, let's look at your newly-added criteria;

1. You're happy to live with loss of drives if more than 1 disk fails - well fair enough, in that case anything but R0 will work for you.
2. You want as much space as possible - given we've ruled out R0 then R5 will clearly give you the most space, followed by R6 then R10.
3. You want good write performance - well R6 is significantly slower than R5 which in turn is slower than R10.
4. If you have a single logical disk array for both the OS and VMs then losing the LD will kill your host, but then so would losing a VM-only logical disk - all your VMs would stop if either LD failed. The difference would be you wouldn't have to reinstall ESX/i again if you only lost a VM-only LD, but then again reinstalling the OS doesn't take long at all, plus it can be backed up. Given you'll lose 600GB if your R1 one them then I'd stick with either R10 or R5 myself.

Solution 2

Again, I can't recommend RAID 0 ever, except in the most unimportant cases.

I second Tom's idea of RAID 1 to mirror the system drive where you install ESX(i), but I don't see the need to RAID-6 your drives, and lose the 2nd drive to parity information.

The reason that RAID-5 is falling out of favor is because of performance (due to the way the information is written across drives) and failure rate (because the likelihood of an unrecoverable read error goes up as drive capacity goes up). RAID-6 doesn't improve performance over RAID-5 when it comes to writes, and your 8 300GB drives aren't anywhere near the capacity where it starts to become statistically significant that you'll fail during a crash.

Solution 3

I always like to have at least two arrays on a server. It's a must for Exchange and SQL for DR purposes.

I would recommend a RAID1/RAID10 setup.

Solution 4

There's quite a lot of articles out there that say that RAID5 is getting to the point of being EOL, because RAID6 is a better alternative, and allows 2 failures before your array collapses and dies.

RAID 10 might also be worth a look (depending on your plan), has higher performance than RAID5/6. You should probably put the ESX server itself on a pair of disks in RAID1, that way your server itself is resilient to a single disk failure. That's where all the clever stuff is. That's what you should protect the most.

I'd go for something like

RAID1: 2x 300GB => ESX Server

RAID6: 1198GB => VM Storage

View more solutions

Comments

[John-ZFS]

hi, many thanks for response disk cost of raid-6 is less than raid-5? also what i wanted to know is when OS disk fails,my data is safe in raid5/6. next when a replacement drive arrives i can put fresh os on it and add the existing datastore, vms to server?

[John-ZFS]

do you recommend RAID-10 all 8 disks = ~1.2TB? at most i need to handle a single disk failure only.

[Chopper3]

R6 is slower at writes that R5.

[TomTom]

Doing something similar here. Hyper-V servers have 4 OS discs in a RAID 10 - that is it. PLENTY of IO etc. Additional discs are used pass through where needed for things like database servers (I dont like the performance overhad of virtual discs here).

[Matt Simmons]

A single RAID-10 would eliminate half of your disk space, which is fine, if you don't mind that, or you need the performance. Heck, you could even RAID-5 the entire thing. It's always a mix between disk space and reliability. Just don't use RAID-0.

[John-ZFS]

hi Tom, could you explain the Hyper-v situation better? looks to me you have RAID-10 for OS (hyper-v) and no RAID for data disks?

[Tom O'Connor]

Fair enough! I always tend to favour separation of OS from data.

[Chopper3]

Oh me too Tom, but with ESX so very little is read or changes in the OS partition (even less so if you're using ESXi) that my normal worries go away, plus the actual OS is in a different partition, just not logical disk with this setup. That said I hate running VMs from local disks - you lose all the nice functions of ESX like vMotion/HA etc.

[wazoox]

RAID-6 is slower than RAID-5 at writing. Both are way slower than RAID-10 anyway. For VMs, RAID-10 is a must.

[Chopper3]

It depends on the VMs though wazoox, we have plenty of non-essential, low-IO VMs running on R6/R5 LUNS just fine.

[John-ZFS]

Hi Chopper, all disks as RAID5: what does it mean when 1 disk fails? ability to recover OS, data? Vs OS and data on separate volumes

[Chopper3]

Maruti, if your disks are already R5 and one disk fails you'll see a significant drop in read performance and an even bigger one for writes, you won't lose data but if you lose a second disk before replacing and rebuilding the first dead disk you will lose all data, if this volume includes the OS then it'll mean an OS reinstall (which is quite quick for ESX/ESXi), if it's for both OS and VMs then it's a reinstall plus VM restore. Hope this helps.

[John-ZFS]

Chopper, thanks for your response. it cleared up things for me. as i am learning wondering how good is RAID-50? does it have any perf improvements over RAID-5?

[Chopper3]

Yes, 50 is faster than 5 as you have two (or more in some cases) banks of R5 'racing' for the data. That said it's a very little used level as R10 usually meets everyone's needs.

[Tom O'Connor]

@Chopper3 Would installing ESX on a USB memory device or CF card be a possible (or sensible) plan? Especially as you said how little it changes.

[Chopper3]

Tom, ESXi was specifically designed to be ran from flash/SD-card - in fact if you buy any HP server you have the option to buy ESXi pre-installed on an SD card - thus saving having to buy disks for SAN-attached servers - we do it all the time, it's called 'embedded ESXi' as opposed to 'installable ESXi'. The embedded version has the manufacturers additional hardware monitoring tools pre-installed but you can still do this yourself if you buy your own SD cards and install it yourself. It actually makes updates easy as you just duplicate an SD card, boot a server from it and change the name/IP.

[John-ZFS]

then how about installing ESXi on USB flash drive and booting from it?

[Chopper3]

Oh yes, sorry, exactly the same, some manufacturers ship their 'embedded' version on SD, some on USB, so long as they're bootable it's the same thing.

[Chopper3]

Oh and by the way, I buy HP BL490c G6's (dual Xeon 55xx/56xx, with 18 x 4GB dimms (the 8GB ones are too expensive), no hard disks and an Emulex 8Gbps FC card) - we install as I mentioned by duplicating an SD card, boot, change name and IP and then apply the server profile via vCenter - it's that easy, we use HP Virtual Connects so have build 16 x ESXi servers in less than two hours from unboxing to built and clustered - isn't technology cool :)