RAID1: How do I "Fail" a drive that's marked as "removed"?

Question

I have (had) a RAID 1 array (2 disk mirror) and one of the disks, sda, failed. So I've replaced the bad disk with a new one, but seem to be stuck on how to get the second drive back up and running as part of the array.

The system is running Ubuntu Server 9.04 and was configured as follows:

MD0 => sda1,sdb1

MD1 => sda3,sdb3

MD2 => sda2,sdb2

 mdadm --detail /dev/md0

shows two drives:

0 /dev/sdb1 "Active Sync"

1 [nothing] "Removed"

MD1 and MD2 look the same.

The tutorial I found says to mark each partition as failed using the command:

mdadm --manage /dev/md0 --fail /dev/sda1

But, since the drive is not there, I get:

mdadm: cannot find /dev/sda1: No such file or directory

Can I skip the failing step? Or is there some other way to fail a partition that's no longer present? Or if I copy the partition table from the good old drive to the new one, will it automatically pick up that it's the replacement?

I'm new at this and don't want to screw it up. :)

Solution 1

You shouldn't need to fail them. Since they should have already been failed when you first noticed the issue and the RAID members are now removed. There are just a few steps to get it back up and running.

1. Setup partitions on the replacement disk. These partitions should be identical in size to that of the failed and currently active disk, and should be marked as partition type "Linux RAID Autodetect" (0xFD). You can simplify this by copying the partition table with sfdisk.

   sfdisk -d /dev/sdb | sfdisk /dev/sda
   

2. If the disk has been used before then you may want to ensure that any existing softRAID information is removed before you begin again.

   mdadm --zero-superblock /dev/sda
   

3. Install an MBR onto the new disk so that it is bootable. Do this from the grub shell. Assumes that /dev/sda is the first disk.

   root (hd0,0)
   setup (hd0)
   quit
   

4. Add new partitions back to the arrays.

   mdadm --add /dev/md0 /dev/sda1
   mdadm --add /dev/md1 /dev/sda3
   mdadm --add /dev/md2 /dev/sda2
   

5. Monitor the status of their reconstruction by viewing /proc/mdstat. You can automate this with.

   watch -n10 cat /proc/mdstat
   

Solution 2

Check http://techblog.tgharold.com/2009/01/removing-failed-non-existent-drive-from.shtml. Use

mdadm /dev/mdX -r detached

Comments

[Nick]

Thanks Dan - Output from the first command included: "Error: sector 0 does not have an msdos signature /dev/sda:unrecognised partition table type" Is that anything to worry about?

[Dan Carley]

It just indicates that the new disk didn't have an existing partition table when sfdisk attempted to read from it. Nothing to worry about. Just double check that the two outputs of fdisk -l /dev/sdb /dev/sda look the same before you proceed.

[Nick]

One small typo: "mdadm --add /dev/md2 /dev/sda1" should be "mdadm --add /dev/md2 /dev/sda2". It looks like it's working- MD0 is marked as [UU] and MD1 says "recovery = 4.2%". So once all are back to [UU] status everything is good? Are there any other steps needed to confirm normal functionality? Thanks!

[Dan Carley]

Whoops - fixed that. Yep, once the arrays are done syncing and all marked UU, then you're back to normal. If the machine is taking a long time to sync and isn't doing anything important then you may wish to speed up the process with the tweaks described here - ducea.com/2006/06/25/… - otherwise just sit back and relax.

[Nick]

Cool, thanks! I adjusted the speed limit/min a little, but I'm not sure it's going any faster. It's Sunday, so I'm in no hurry. :) If I wanted to add a third mirrored drive, can I just repeat the steps above, substituting sdc for sda?

[Dan Carley]

You can't throw a 3rd disk in as an active mirror because RAID1's functionality is limited to 2 array members. It would be used as a spare instead.

[Nick]

Looks like the arrays are back in sync, but it won't boot off of the new drive. It goes to a single blinking cursor in the top left corner when grub is supposed to load. If I set the other drive as the first boot device it boots fine. Do I need to set a bootable flag on the new drive?

[Chopper3]

Well done on you 10K Dan, good timing :) best wishes, Phil.

[Dan Carley]

Nick> Ah, I've added step #3. That should do it. Phil> Thanks! :)

[Scott Pack]

Welcome to Server Fault! Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference.

[bbonev]

+1 - the proposed command solved my problem too