How many disks is too many in this RAID 5 configuration?

Question

• HP 2012i SAN,
• 7 disks in RAID 5 with 1 hot spare,
• took several days to expand the volume from 5 to 7 300GB SAS drives.

Looking for suggestions about when and how I would determine that having 2 volumes in the SAN, each one with RAID 5, would be better??

I can add 3 more drives to the controller someday, the SAN is used for ESX/vSphere VMs.

Thank you...

Solution 1

I've wrestled with this question for a while. There are a number of factors determining how many disks should go into a RAID5 array. I don't know the HP 2012i, so here is my generic advice for RAID5:

• Non-recoverable read error rate: When a non-recoverable read error occurs, that read fails. For a healthy RAID5 array this is no problem since the missed read can be found in the parity information. If one happens during a rebuild, when the entire RAID5 set is read in order to regenerate the parity info, it can cause the entire RAID5 array to be lost. This rate is measured like this: "1 per 10¹⁴ bits" and is found on the detail tech-specs for drives. You do not want your RAID5 array to be any more than half that size. Enterprise drives (10K RPM SAS qualifies) can go longer than Desktop drives (SATA).
  • For an example of this spec, Seagate Barracuda ES.2 data-sheet.
• Performance degradation during rebuilds: If performance noticeably sucks during rebuilds, you want to make sure your array can rebuild quickly. In my experience write performance tends to suck a lot worse during rebuilds than reads. Know your I/O. Your tolerance for bad I/O performance will put an upper limit on how large your RAID5 array can get.
• Performance degradation during other array actions: Adding disks, creating LUNs, changing stripe widths, changing RAID levels. All of these can impact performance. Some controllers are very good about isolating the performance hit. Others aren't so good. Do some testing to see how bad it gets during these operations. Find out if restriping a 2nd RAID5 array impacts performance to the first RAID5 array.
• Frequency of expand/restripe operations: Adding disks, or on some controllers creating new LUNs, can also cause the entire array to redo parity. If you plan on active expansion, then you'll be running into this kind of performance degradation much more often than simple disk failure-rate would suggest.

RAID6 (double parity) is a way to get around the non-recoverable read error rate problem. It does increase controller overhead though, so be aware of the CPU limits on your controllers if you go there. You'll hit I/O bottlenecks faster using RAID6. If you do want to try RAID6, do some testing to see if it'll behave the way you need it to. It's a parity RAID, so it has the same performance penalties for rebuilds, expansions, and restripes as RAID5, it just lets you grow larger in a safer way.

Solution 2

There isn't really a clear limit on the number of disks in RAID 5 per se. The limits you'll meet are typically related to RAID 5's relatively poor write performance, and limitations imposed elsewhere (RAID controller, your own organization of data etc).

Having said that, with 7-8 disks in use you're close to upper bound on the common RAID 5 deployments. I'd guesstimate that the wast majority of RAID 5 deployments are using <10 disks. If more disks are wanted one would generally go for nested RAID levels such as RAID "50".

I'm more puzzled by your choice to keep one big array for all this. Would your needs not be better served by 2 arrays, one RAID5 for slow, mostly read data, and one RAID 10 for more I/O intensive data with more writes?

Solution 3

How much space do your NEED? You currently have ~1.995TB, could you live with ~1.140TB? if so then I'd strongly suggest you move to RAID 10 - not only will it be faster but you'll be able to lose half your disks without user impact. If you choose to go this way I'd order the extra 4 x 300GB disks now and build it with all 12 disks on day one rather than add them later - this will also handily give you ~2.280TB available too.

Solution 4

For my money, I'd do two three-disk arrays, with one shared hot spare.

If you don't have an need for a single block of space larger than a 3 disk array, then there is no reason to cram all 6 disks into 1 raid. You're not going to gain anything in performance or over-all space, and, given a two disk failure, you're probably going to be in a better place.

@Dayton Brown: The total space will be the same...3 1TB drives in a RAID5 is what, 1.8TB? 6 will be 3.6 by the same measure, so in that sense you'll have more space in that particular RAID volume, even though the total space will remain the same. The difference is, RAID5 only allows for 1 parity drive, whether there are 3 drives in the RAID or 300, so splitting the drives into manageable groups adds protection against multiple failures. Even if you lost 3 disks, for example, you'd only lose half of your data.

If you moved to RAID6, a six disk array would make more sense, because you could lose two and be okay. But most people jump straight to RAID10, and skip 6.

Solution 5

Any number of disks in a RAID 5 array is too many disks. The only possible exception maybe if you want the maximum storage/$ for something where speed and a reasonably quick recovery from failure is not an issue.

You've invested in expensive hardware to build a SAN; arrays, switches, fibre, training etc. Then you've put in pretty fast (relatively) expensive disks. Furthermore you want to put VMs on it. With this investment and usage it looks like you want to make it as fast as you can so your VM users don't complain about performance. So why touch RAID 5 at all?

BAARF is old, but still relevant as using RAID5 is as bad an idea now as it was a decade ago.

View more solutions

Comments

[Joe]

I suspect that the poster recommends no more than 3 because if you lose another disk (or encounter disk errors of almost any kind) during the rebuild, the entire array may be lost. That's why I said below that it is preferable to keep the disk count as low as possible, while offering the necessary space.

[Satanicpuppy]

+1: The rebuild is a sticking point. It is not at all uncommon to lose a drive while you're rebuilding. I once lost 19 drives in a 24 disk cabinet (8 RAID5 arrays, no hot spares), one after the other. The rebuilds stressed them, and backing up the data to neighboring arrays stressed THOSE, and THEY started failing...Scary stuff.

[Rob Moir]

+1 to sysadmin + SatanicPuppy. Especially Satanicpuppy's point - if you buy a big array and fill it with disks in one go/ bulk purchase, they will stand a higher chance of being the same age and possibly the same batch. When one goes the others may well not be far behind.

[Philip]

+1, General rule of thumb 7 or 8 dirves for RAID5; and about double that for RAID6 (though your card's performance will likely become a bottleneck). After that go RAID 50 or 60.

[silvesthu]

A 6 x 1TB raid5 array will be closer to 4.6 TB, not 3.6TB

[Satanicpuppy]

+1: Yep yep. You're right. 1 parity disk out of 6 instead of 1 out of 3. My bad.

[Dayton Brown]

@Joe. Thanks for the explanation. That's a very good reason.

[Whisk]

There's a very good discussion of this here: blogs.zdnet.com/storage/?p=162