3-way RAID 1 vs. 2-way RAID 1 + hot spare

Question

Assuming the following parameters for a new server storage configuration:

• All drives are SSD
• Few large drives are preferred over many small drives
• RAID 1/mirroring is preferred over RAID 5/6/parity
• A dedicated hardware RAID adapter is preferred over software RAID or storage spaces
• Edit: Only one single RAID array is required

So to me it all boils down to a three drive setup with either

• 2-way RAID 1, with hot spare
• 3-way RAID 1, no hot spare

While the obvious advantages of the 3-way setup are

• Potential read performance bonus
• Concurrent loss of two drives supported

Are there any points to make in favor of the 2-way hot spare setup except maybe these?

• All data is also written to the third drive, wearing it out like just like the other two, while a hot spare won't be written to before a potential array rebuild.
• Potential write performance penalty due to all data needing to be written three times opposed to twice (which could be negated by the hardware RAID controller)

Edit: Added parameter to clarify that only one single RAID array is desired

Solution

Edit: Added parameter to clarify that only one single RAID array is desired

In that case you get no benefits. The purpose of a hot spare nowadays is to serve many different arrays/pools. If you have 12 pools out of 24 disks, you just need one or two spares instead of 12 for first response until someone can be at the server and replace the disks.

All data is also written to the third drive, wearing it out like just like the other two, while a hot spare won't be written to before a potential array rebuild.

In my eyes this is a downside, because you only know if sectors are bad on your drive (applies more to hard disks than SSDs) when you start writing to it, which will be the most dangerous and stressful time (for the remaining drive as well as the general load as well as your sanity. Discovering errors just then is something you would like to avoid - predictability is valued higher than less wear and tear. I would make exceptions for things like write-heavy cache devices that wear out the SSDs very fast, or an array of cheap SSDs that should last the maximum amount of time.

Potential write performance penalty due to all data needing to be written three times opposed to twice (which could be negated by the hardware RAID controller)

As long as you have remaining bus and controller bandwidth, it should not interfere. On the other hand, now you can read from three disks instead of two, meaning roughly 50% performance increase (less actually, but this is the general idea). Exceptions apply if the third disk is very slow, but I assume they are similar or identical types.

Comments

[Hannes]

Actually that came to my mind, too. In my eyes it's quite risky to rely on a disabled or at least idle spare device which might turn out malfunctional when a rebuild is required - during which the array is not protected. Initially, I just wanted to mention the potential higher cost of wearing out all devices, but I totally agree that predictability is much higher valued than less wear.

[user121391]

@Hannes For HDDs, it is pretty clear because wearout is a very small theoretical problem, while for SSDs it is much more interesting. But even the SSD TBW values are conservative to reduce warranty claims, so you might add some additional padding after the disk is beyond its predicted lifespan... but then all you pool disks would have the same problem, so I guess it is more a question of perspective: are you interested in general pool health or on the negative effects on the spare itself.

[Hannes]

Since the I/O profile is expected to be rather read heavy, I would assume that the drives won't reach their TBW before the end of their warranty lifecycle. Given that enterprise SSD models that come into consideration provide >3,000 TBW as of today, that won't be an issue. It might be an issue for heavy write profiles, I agree. You'd have to swap drives here from time to time, but I'd still prefer resiliency over the hot spare drawbacks.