How to bring up an inactive mdadm RAID after a reboot while adding a drive to a raid-6?

opensuse raid mdadm webmin

12,639

If mdadm cannot find all the parts of an array when assembling it, it won't automatically activate it for use. You can manually activate it with the command mdadm --run /dev/md3. It may start up in read-only mode, in which case you can put it in read-write mode with mdadm --readwrite /dev/md3.

This is a safe operation: if there aren't enough pieces of the array available to have a complete set of data (eg. three drives missing from a RAID6, or a matched pair missing from a RAID10), mdadm will do nothing.

12,639

moxi

Updated on September 18, 2022

Comments

moxi over 1 year

I have been using webmin to manage my RAID array for several years, and I never had any reason to use the mdadm commands in the terminal.

However, some days ago, I added a disk to my RAID6 array of six drives and there was no progress in the reshape. I assumed if I reboot the computer, the problem would be solved, but after reboot I am in a bigger problem now.

Although the array is in a clean state after reboot, I cannot start the array on /dev/md3. See the commands and attached webmin screen shots.

250:~ # cat /etc/os-release
NAME="openSUSE Tumbleweed"
# VERSION="20161112"
ID=opensuse
ID_LIKE="suse"
VERSION_ID="20161112"
PRETTY_NAME="openSUSE Tumbleweed"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:tumbleweed:20161112"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"


250:~ # uname -a
Linux 250 4.8.6-2-default #1 SMP PREEMPT Thu Nov 3 13:00:34 UTC 2016 (1d89b44) x86_64 x86_64 x86_64 GNU/Linux


250:~ # rpm -q webmin 
webmin-1.820-1.noarch


250:~ # mdadm -V
mdadm - v3.4 - 28th January 2016


250:~ # cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md127 : active raid6 sdi2[7] sdf2[4] sdc2[2] sdb2[0] sda2[1] sdd2[5] sdg2[6]
      5119360 blocks super 1.0 level 6, 128k chunk, algorithm 2 [7/7] [UUUUUUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md1 : active raid6 sdc3[2] sdf3[5] sda3[1] sdb3[3] sdd3[4] sdi3[6] sdg3[0]
      153599360 blocks super 1.0 level 6, 128k chunk, algorithm 2 [7/7] [UUUUUUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md3 : inactive sdi4[7] sdf4[4] sdd4[5] sdb4[0] sdc4[2] sda4[1] sdg4[6]
      3103718444 blocks super 1.0

md0 : active raid6 sdf6[5] sdc6[2] sdb6[4] sda6[1] sdi6[8] sdd6[6] sdg6[7]
      7149870080 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/7] [UUUUUUU]
      bitmap: 0/11 pages [0KB], 65536KB chunk

md2 : active raid6 sdb5[0] sdc5[2] sdd5[5] sda5[1] sdi5[7] sdf5[4] sdg5[6]
      5119993600 blocks super 1.0 level 6, 128k chunk, algorithm 2 [7/7] [UUUUUUU]
      bitmap: 0/8 pages [0KB], 65536KB chunk

unused devices: <none>


250:~ # mdadm -D /dev/md3
/dev/md3:
        Version : 1.0
  Creation Time : Sat Dec 12 07:16:21 2015
     Raid Level : raid6
  Used Dev Size : 443387648 (422.85 GiB 454.03 GB)
   Raid Devices : 7
  Total Devices : 7
    Persistence : Superblock is persistent

    Update Time : Thu Nov 17 14:17:34 2016
          State : active, Not Started 
 Active Devices : 7
Working Devices : 7
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 128K

  Delta Devices : 1, (6->7)

           Name : any:3
           UUID : 51cc1190:dc4821a5:c1161e00:b73e02ec
         Events : 139752

    Number   Major   Minor   RaidDevice State
       0       8       20        0      active sync   /dev/sdb4
       1       8        4        1      active sync   /dev/sda4
       2       8       36        2      active sync   /dev/sdc4
       4       8       84        3      active sync   /dev/sdf4
       6       8      100        4      active sync   /dev/sdg4
       5       8       52        5      active sync   /dev/sdd4
       7       8      132        6      active sync   /dev/sdi4


250:~ # mdadm -E /dev/sd[abcdfgi]4
/dev/sda4:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x5
     Array UUID : 51cc1190:dc4821a5:c1161e00:b73e02ec
           Name : any:3
  Creation Time : Sat Dec 12 07:16:21 2015
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 886775528 (422.85 GiB 454.03 GB)
     Array Size : 2216938240 (2114.24 GiB 2270.14 GB)
  Used Dev Size : 886775296 (422.85 GiB 454.03 GB)
   Super Offset : 886775792 sectors
   Unused Space : before=0 sectors, after=480 sectors
          State : clean
    Device UUID : 92f5e8d6:4e55facc:5446de77:ee298eee

Internal Bitmap : -16 sectors from superblock
  Reshape pos'n : 0
  Delta Devices : 1 (6->7)

    Update Time : Thu Nov 17 14:17:34 2016
  Bad Block Log : 512 entries available at offset -8 sectors
       Checksum : 5c4a0af8 - correct
         Events : 139752

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 1
   Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb4:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x5
     Array UUID : 51cc1190:dc4821a5:c1161e00:b73e02ec
           Name : any:3
  Creation Time : Sat Dec 12 07:16:21 2015
     Raid Level : raid6
   Raid Devices : 7


 Avail Dev Size : 886775528 (422.85 GiB 454.03 GB)
     Array Size : 2216938240 (2114.24 GiB 2270.14 GB)
  Used Dev Size : 886775296 (422.85 GiB 454.03 GB)
   Super Offset : 886775792 sectors
   Unused Space : before=0 sectors, after=480 sectors
          State : clean
    Device UUID : b36bd7a1:ec792d23:51c0e852:e9ba91b5

Internal Bitmap : -16 sectors from superblock
  Reshape pos'n : 0
  Delta Devices : 1 (6->7)

    Update Time : Thu Nov 17 14:17:34 2016
  Bad Block Log : 512 entries available at offset -8 sectors
       Checksum : 1f79aeed - correct
         Events : 139752

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 0
   Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc4:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x5
     Array UUID : 51cc1190:dc4821a5:c1161e00:b73e02ec
           Name : any:3
  Creation Time : Sat Dec 12 07:16:21 2015
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 886783720 (422.85 GiB 454.03 GB)
     Array Size : 2216938240 (2114.24 GiB 2270.14 GB)
  Used Dev Size : 886775296 (422.85 GiB 454.03 GB)
   Super Offset : 886783984 sectors
   Unused Space : before=0 sectors, after=8672 sectors
          State : clean
    Device UUID : ab40e58e:724de1e3:e502ccf0:831a93cd

Internal Bitmap : -16 sectors from superblock
  Reshape pos'n : 0
  Delta Devices : 1 (6->7)

    Update Time : Thu Nov 17 14:17:34 2016
  Bad Block Log : 512 entries available at offset -8 sectors
       Checksum : 8320399c - correct
         Events : 139752

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 2
   Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd4:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x5
     Array UUID : 51cc1190:dc4821a5:c1161e00:b73e02ec
           Name : any:3
  Creation Time : Sat Dec 12 07:16:21 2015
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 886775528 (422.85 GiB 454.03 GB)
     Array Size : 2216938240 (2114.24 GiB 2270.14 GB)
  Used Dev Size : 886775296 (422.85 GiB 454.03 GB)
   Super Offset : 886775792 sectors
   Unused Space : before=0 sectors, after=480 sectors
          State : clean
    Device UUID : b0f9f68d:0ad6f39a:50043287:10b0570d

Internal Bitmap : -16 sectors from superblock
  Reshape pos'n : 0
  Delta Devices : 1 (6->7)

    Update Time : Thu Nov 17 14:17:34 2016
  Bad Block Log : 512 entries available at offset -8 sectors
       Checksum : f6ed233 - correct
         Events : 139752

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 5
   Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf4:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x5
     Array UUID : 51cc1190:dc4821a5:c1161e00:b73e02ec
           Name : any:3
  Creation Time : Sat Dec 12 07:16:21 2015
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 886775528 (422.85 GiB 454.03 GB)
     Array Size : 2216938240 (2114.24 GiB 2270.14 GB)
  Used Dev Size : 886775296 (422.85 GiB 454.03 GB)
   Super Offset : 886775792 sectors
   Unused Space : before=0 sectors, after=480 sectors
          State : clean
    Device UUID : c63684ba:59abf1dc:603d3c94:b5efd4b6

Internal Bitmap : -16 sectors from superblock
  Reshape pos'n : 0
  Delta Devices : 1 (6->7)

    Update Time : Thu Nov 17 14:17:34 2016
  Bad Block Log : 512 entries available at offset -8 sectors
       Checksum : 34815d4d - correct
         Events : 139752

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 3
   Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg4:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x5
     Array UUID : 51cc1190:dc4821a5:c1161e00:b73e02ec
           Name : any:3
  Creation Time : Sat Dec 12 07:16:21 2015
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 886775528 (422.85 GiB 454.03 GB)
     Array Size : 2216938240 (2114.24 GiB 2270.14 GB)
  Used Dev Size : 886775296 (422.85 GiB 454.03 GB)
   Super Offset : 886775792 sectors
   Unused Space : before=0 sectors, after=480 sectors
          State : clean
    Device UUID : 9ef6c9e1:5a56c716:4cc8645f:bd92060b

Internal Bitmap : -16 sectors from superblock
  Reshape pos'n : 0
  Delta Devices : 1 (6->7)

    Update Time : Thu Nov 17 14:17:34 2016
  Bad Block Log : 512 entries available at offset -8 sectors
       Checksum : b4f6f61a - correct
         Events : 139752

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 4
   Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi4:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x5
     Array UUID : 51cc1190:dc4821a5:c1161e00:b73e02ec
           Name : any:3
  Creation Time : Sat Dec 12 07:16:21 2015
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 886775528 (422.85 GiB 454.03 GB)
     Array Size : 2216938240 (2114.24 GiB 2270.14 GB)
  Used Dev Size : 886775296 (422.85 GiB 454.03 GB)
   Super Offset : 886775792 sectors
   Unused Space : before=0 sectors, after=480 sectors
          State : clean
    Device UUID : a6046dca:1edf6818:5fe92892:2e3a966f

Internal Bitmap : -16 sectors from superblock
  Reshape pos'n : 0
  Delta Devices : 1 (6->7)

    Update Time : Thu Nov 17 14:17:34 2016
  Bad Block Log : 512 entries available at offset -8 sectors
       Checksum : 368f556c - correct
         Events : 139752

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 6
   Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)


250:~ # mdadm --examine --scan
ARRAY /dev/md/0  metadata=1.2 UUID=ad2e4877:bb799c91:49e20b58:c0866104 name=430:0
ARRAY /dev/md/swap  metadata=1.0 UUID=a8f1a9d5:bd89d7db:0764ba4d:5a515066 name=any:swap
ARRAY /dev/md/2  metadata=1.0 UUID=0d675c76:d91e6dd2:b8b89871:5af91fed name=any:2
ARRAY /dev/md/3  metadata=1.0 UUID=51cc1190:dc4821a5:c1161e00:b73e02ec name=any:3
ARRAY /dev/md/0  metadata=1.0 UUID=8cedcb83:66499024:f9d66cca:7e7b367b name=any:0

moxi over 7 years

md127 is just a name of an array and it is ok. I have run mdadm --examine --scan. This is the out put:
frostschutz over 7 years

What's in dmesg when you assemble it?
moxi over 7 years

The array cannot be assembled when it is stopped, it is only a reboot that assembles the array. The dmesg after a reboot is [ 39.576104] md/raid:md3: reshape_position too early for auto-recovery - aborting. [ 39.576110] md: pers->run() failed ...
moxi over 7 years

Problem solve by recreating the superblock
moxi over 7 years

using mdadm --create /dev/md3 -l6 -n6 -c128 --layout=ls --metadata=1.0 --assume-clean /dev/sdb4 /dev/sda4 /dev/sdc4 /dev/sdf4 /dev/sdg4 /dev/sdd4 . The sequence of the drives is obtained from the output of ` mdadm -D /dev/md3`.