Disk failed part way through 3ware RAID 5 rebuild

Question

I have a 3ware 9650SE RAID controller with a RAID 5 array containing 15 Seagate ST31000340NS disks. After noticing ECC errors in the Port 10 drive I replaced it with a spare and began a RAID rebuild. Part way through the rebuild the Port 5 disk failed completely, which rendered the array inoperable because the new disk in Port 10 was incomplete. The array remained in use during the rebuild until the failure of the Port 5 disk. I hoped to recover the data by putting back the original Port 10 disk, but the RAID controller did not add it back to the array. Instead, it was listed as "available". My question is, how can I force the controller to recognise the original Port 10 disk in its original location? There is no "add disk" option in the 3dm2 interface.

* EXTRA INFO * Thanks for all the comments and suggestions relating to my original posting. I should have mentioned before that the array was mounted read-only during the rebuild. I don't know if that makes any difference to the chances of forcing the controller to accept the original disk back. There isn't a backup by the way. Whatever happens, I have certainly learned my lesson re. RAID5.

Solution 1

I believe you are out of luck. This is one of the dangers of RAID5. Since the array was in use, all the other disks are now out of sync with the original port 10 disk.

updated: Regarding the update read-only mounting...Whether or not this works is really going to be an implementation detail of the 3ware. Even if you mounted read-only, the raid controller could have updated some metadata on the disks and decided this configuration is not recoverable. That's what I would expect.

Solution 2

Your best option is to rebuild from backup. Since the array was in use, the data would be out of sync on the 10 disk.

RAID 5 is no longer really being recommended for use as drive sizes grow larger; the odds of an unrecoverable error on the drives are increasing, and aren't typically found until you have a disk fail on the RAID 5 array (which is when the second disk and it's latent bad spot is found).

Solution 3

You may be lucky if the error on the second drive is in an portion of the disk that is unused by the file system. So if you do not have any backups, you could try rebuilding with the "ignore ECC errors on rebuild" flag set. You would then want to run some form of consistency check over your file system and you may have to expect some data corruption in the worst case. Still, it may be preferrable to losing the entire volume.

Comments

[Oskar Duveborn]

+1 Yeah the array is likely waste, restore it from backups instead - and if uptime is that important, consider adding a hotspare to your RAID5 array or going RAID6.

[Joe]

This also perfectly illustrates the problem with using lots of disks in a RAID5 array; if you lose a disk, your array is vulnerable to a total irrecoverable data loss if anything bad happens during rebuild.

[Dan]

Thank you all for your comments and suggestions. I have certainly learned my lesson re. RAID5. There isn't a backup unfortunately. I did, however, take the precaution of mounting the array read-only before commencing the rebuild. I should have mentioned that before -sorry. Does that make a difference to the chances of recovering the data? -Dan.

[Dan]

I just realise I forgot to post an update to my problem last year. Several weeks after the incident I was able to recover the array using a script provided by 3ware technical support. It was customised for my array by 3ware using information dumped by the lsigetlunix.sh script. This recovery was only possible because the data had been mounted read-only while the rebuild was taking place, but I was still incredibly lucky to recover it all. I later converted that server to RAID6. Thanks again to everyone who responded to my original question.