mdadm, RAID5 all disks marked as spare, won't start

Question

RAID5 config with mdadm and 4 disks

After some incidents, eg one of the drives was lost, had to reboot the server and rebuild the array, my raid is in a funny situation

cat /proc/mdstat

Personalities : 
md0 : inactive sdb[1](S) sda[4](S) sdd[3](S) sdc[2](S)
      7814057984 blocks

unused devices: <none>

mdadm -E /dev/sd[a-e] | grep Event

     Events : 946860
     Events : 946860
     Events : 946848
     Events : 946860

mdadm -D /dev/md0

mdadm: md device /dev/md0 does not appear to be active.

mdadm --assemble /dev/md0 /dev/sda /dev/sdb /dev/sdc /dev/sdd --force --verbose

mdadm: looking for devices for /dev/md0
mdadm: /dev/sda is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdb is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 3.
mdadm: ignoring /dev/sdb as it reports /dev/sda as failed
mdadm: ignoring /dev/sdc as it reports /dev/sda as failed
mdadm: ignoring /dev/sdd as it reports /dev/sda as failed
mdadm: no uptodate device for slot 0 of /dev/md0
mdadm: no uptodate device for slot 1 of /dev/md0
mdadm: no uptodate device for slot 2 of /dev/md0
mdadm: no uptodate device for slot 3 of /dev/md0
mdadm: added /dev/sda to /dev/md0 as 4
mdadm: /dev/md0 assembled from 0 drives and 1 spare - not enough to start the array.

I know that one of my drives, /dev/sdc is about to fail as I get SMART reports for bad sectors (7 bad sectors)

Solution

Try assembling it without the odd third disk sdc, i.e.

mdadm --assemble /dev/md0 /dev/sda /dev/sdb /dev/sdd --verbose

That sounds like it could work because the remaining three appear to be in sync and with RAID-5, N-1 disks is sufficient to restart the array in degraded mode.

It's possible that the device indices aren't right, examine mdadm -E output and see if you can identify the set of three working disks. From the error messages, it sounds like both sdc and sda had failed simultaneously at some point, which is something RAID-5 can't handle gracefully.

(Originally I had suggested to omit the third disk by replacing it with the string missing, but that is --create syntax as pointed out by S.Haran below.)

Afterwards, after you verify things are in order, you can try to re-add the third (fourth) disk with:

sudo mdadm /dev/md0 --add /dev/sdc

Comments

[AWippler]

Did you remove the broken sda from the raid array?

[DimitrisD]

no, nothing has been removed yet

[nit17]

I'm experiencing a similar issue when one of my backplanes magically died and all disks in the array are marked as Spare. However, in my case the "Event" numbers are all identical and no actual disks have failed. Weird.

[Josip Rodin]

Thanks for the note about missing. With the option --assemble, that placeholder indeed isn't used, that is, implicit omissions are fine.