mdadm Stuck at rebuilding spare

Question

I have the following Raid configuration,

cat /proc/mdstat 
Personalities : [raid1] [raid6] [raid5] [raid4] 
md4 : active (auto-read-only) raid5 sda5[0] sdd5[4] sdc5[2] sdb5[1]
      1536 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UUU_]

md3 : active raid5 sda4[0] sdd4[4] sdc4[2] sdb4[1]
      11587487232 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]

md2 : active raid5 sda3[0] sdd3[4] sdc3[2] sdb3[1]
      104756736 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]

md1 : active raid1 sda2[0] sdd2[3] sdc2[2] sdb2[1]
      1048000 blocks super 1.2 [4/4] [UUUU]

md0 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
      8384448 blocks super 1.2 [4/4] [UUUU]

unused devices: 

mdadm tells me it is rebuilding for it has been like this at least a week,

mdadm --detail /dev/md4
/dev/md4:
        Version : 1.2
  Creation Time : Fri Mar 28 10:17:36 2014
     Raid Level : raid5
     Array Size : 1536
  Used Dev Size : 512
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Fri Mar 28 10:19:54 2014
          State : clean, degraded 
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

           Name : rescue:4
           UUID : 9cc47e82:14504b6d:0f442b8e:a3fc1c04
         Events : 12

    Number   Major   Minor   RaidDevice State
       0       8        5        0      active sync   /dev/sda5
       1       8       21        1      active sync   /dev/sdb5
       2       8       37        2      active sync   /dev/sdc5
       4       8       53        3      spare rebuilding   /dev/sdd5

How can I figure out more information about what is going on? Is it stuck is it rebuilding?

EDIT: This is happening on a new server 4x4tb I have been copiying couple tb's of data to it.

fstab on the server

proc /proc proc defaults 0 0
/dev/md/0 none swap sw 0 0
/dev/md/1 /boot ext3 defaults 0 0
/dev/md/2 / ext4 defaults 0 0
/dev/md/3 /home ext4 defaults 0 0
/dev/md/4 /home ext3 defaults 0 0

Solution

md4 : active (auto-read-only) raid5 sda5[0] sdd5[4] sdc5[2] sdb5[1]
      1536 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UUU_]

It says right there - it's in read-only mode. Switch it to read-write mode:

mdadm --readwrite /dev/md4

By the way, this isn't a large array. It's a tiny slice of your 4TB disks. Bit of a wacky setup as well, if you ask me.

Comments

[Nathan C]

RAID-5 is notorious for being really, really slow. 1.5TB you can expect 2-3 days to rebuild.

[Hamza Yerlikaya]

@NathanC I made a typo I meant to say a week, but it is a 4 tb disk. Is this the normal raid 5 behavior. This is happening on a new server 4x4tb I have been copying couple tb's of data to it.

[Nathan C]

Are you actively moving data around? This slows rebuilds considerably.

[Hamza Yerlikaya]

Can you give some more information about the situation? or Where can I read/find more about why it happened?

[Hamza Yerlikaya]

I thought md4 was home but now checking fstab (included in the question) home is both md3 and md4 as for raid structure I have not built this server I am trying to figure out what is going on.

[Steve Townsend]

"Where can I read/find more about why it happened?" → Your logs.