RAID-6: better to replace two dead drives at the same time, or one at a time?

Question

We have a 16-drive RAID-6 that has three problem drives. Two are already dead, and the third is giving SMART warnings. (Nevermind how it got in such a bad state.)

Obviously we want to replace the dead drives before the one that is still working, but is it better to:

1. replace one dead drive, let the RAID rebuild, then replace the other, and let it rebuild again; or

2. replace both drives at once and let it rebuild both in parallel?

To put it another way, will we get back to a state of redundancy faster by reintroducing one drive or two? Does rebuilding two drives in parallel slow the rebuild process?

In case it matters, the controller is a 3ware 9650SE-16ML.

Solution 1

!!!!! ONE !!!!!

Do one at a time, seriously dude, don't think of doing this ANY other way ok.

Anything else will test your full system restoration skills.

Solution 2

Do you have good, recent backups? If not do you think you can get them in reasonable time?

I'd honestly be more concerned about tripping the bad drive offline during a rebuild than anything else - If you're already throwing SMART errors you're more than halfway there.

My suggestion would be to confirm your backups, then rebuild one drive at a time to try to recover to a state where you can replace the one throwing SMART errors (dead drives first, soft-errors last).

If you have no backups it's a crap shoot: Backing up may create enough soft errors to mark the marginal drive as failed, as may trying to do a rebuild.

Comments

[user9517]

Cross everything you got that can be crossed and send your favourite $deity a large donation!

[Chopper3]

Can I just ask one question regarding this; can you let us know the EXACT make and model of disk in this array please - if my suspicions are correct you may very well see this question become a useful referal point for future users asking certain questions. Thank you.

[Warren Young]

I don't know the exact model, and can't query the controller to find out, because someone else decided the machine should be turned off until the replacement drives get there. From data captured before then, I can say they're 1 TB Seagates with serials beginning with 9QJ.

[voretaq7]

@Warren - AIEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE! They turned it off? Bad JuJu my friend! It's too late this time, but generally speaking shutting the drives down (especially if they're older drives that have been up and running a long time) gives them an opportunity to throw errors on spinup (and cause the controller to say "Yeah, that drive is fried too now")

[Tch]

@voretaq7: I've once sent all the disks from an MSA-20 box to hell by shutting it down after ~3 years of uptime and continuous use. Will never do that again :-)

[Warren Young]

The array is up and rebuilding now, so I can get exact models, for anyone who cares. The original hard drives were ST31000340NS, meaning they're the server-rated version of the one Chopper3 was asking about. So are these notorious for failing or something? (The new ones are ST31000524NS.)

[Warren Young]

Most or all of the data on this array is a kind of cache, to avoid the need to pull terabytes of data repeatedly through a slow link. This cache data is all replaceable, either by downloading it again over months (once) or by shipping it to a site that can copy from another array. So, backups aren't the issue. What we're trying to prevent by saving the array is the days to weeks of downtime shipping the server to a service depot, re-populating the array, and shipping it back.

[voretaq7]

in that case, what @chopper3 said is pretty much The Law Of The Land: Rebuild one drive at a time and PRAY REALLY REALLY HARD that you don't trip the marginal drive offline with the extra read load.

[Chopper3]

Phew - glad to hear it.

[voretaq7]

The two items I would add to this answer are (1) PRAYER (to whatever deity you like) and (2) MONITORING once you get everything back to a safe state (so you'll know when drives fail in the future and can address the problem before you have two and a half failures. Optionally you can also configure a hot spare in the array for the future.

[Warren Young]

Re: prayer, no comment. :) Re: monitoring, I've been advocating that for years; maybe this will light a fire under someone. Re: RAID-10, too much data in a bid market; when 3 TB disks came out, we didn't triple redundancy, we cut the number of disks by 1/3. Sigh. Re: hot spares, we do that now that drives are big enough to allow it, but this particular server was 16 drives in a 16 drive enclosure, when 1 TB drives were the biggest you could get, and we really needed all 14 available TB. Going to a 24-drive system wouldn't have worked; see previous. :)

[pfo]

It might not even be a good idea to replace anything yet as a rebuild might discover/trigger a media error (possibly on the SMART indicated disk).

[ddm-j]

If the drives have already failed, then there is no reason to keep them around -- rather, I'd expect that two consecutive rebuilds are more stressful for the other drives than a single one.

[Joel Coel]

+1, This. While two consecutive rebuilds add more stress and more likely to cause the third drive to fail before you finish both, it's also a faster rebuild, and if the marginal drive fails while rebuilding the 2nd disk, you still stay online. So the fastest, safest way to a fault-tolerant state is one at a time.

[Lily Chung]

I've upvoted this answer, which I agree is the most conservative approach. However, in principle, replacing two disks at once could be less stressful on the remaining drives: each RAID-6 stripe will have enough redundancy to reconstruct data for any two failed drives, and so replacing two at once could result in only a single scan of the surviving drives. Replacing one at a time guarantees two scans, but with increased safety sooner. All of this depends on implementation details, though, so I still recommend one drive at a time.

[poige]

Absolutely pointless answer: no reasoning, just "do that", and traditionally for QA sites bunch of upvotes. Seriously.