Ubuntu Server 12.04, MDADM device number suddenly changes?

Question

I'm facing a really annoying bug, I have set up a Software RAID1 with mdadm, and the setup went just ok with this results

/dev/md/lucas.mgscreativa.com.ar:0

I have also configured mdadm.conf like this

DEVICE /dev/sda /dev/sdb

# auto-create devices with Debian standard permissions CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts MAILADDR root

# definitions of existing MD arrays ARRAY /dev/md/lucas.mgscreativa.com.ar:0 metadata=1.2
name=lucas.mgscreativa.com.ar:0
UUID=c913486a:e62c7ea1:cfb98b6b:253d1f62

And fstab configured like this

/dev/md/lucas.mgscreativa.com.ar:0 /media/data     ext4
defaults,noatime      0       0

But, some day, my array name suddenly changed from /dev/md/lucas.mgscreativa.com.ar:0 to /dev/md0, so I have done some changes in fstab and mdadm.conf (of course, after hours and hours of tryal and error) and everything went just ok, but today, AGAIN, the MD number changed from /dev/md0 to /dev/md127!!!

What's going on here? is this a bug I guess.

Is there any way to fix the MD number so my server can work as it shhould, without flaws?

Thanks!

Solution 1

First: you are probably having errors in your array. You need to check its status, immediately.

Example:

root@locutus:~# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Fri Jun  1 15:30:31 2012
     Raid Level : raid10
     Array Size : 1953119232 (1862.64 GiB 1999.99 GB)
  Used Dev Size : 976559616 (931.32 GiB 1000.00 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sun Nov 25 21:39:44 2012
          State : clean 
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 512K

           Name : locutus:0  (local to host locutus)
           UUID : b81ac63a:51a9b01c:c01812ec:86c534c4
         Events : 182

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       49        1      active sync   /dev/sdd1
       2       8       17        2      active sync   /dev/sdb1
       3       8       33        3      active sync   /dev/sdc1

You will very likely find that you have errors... those will need to get addressed.

Moving on, you should change the way your fstab is set up, to use UUID instead of using devicename. Use the blkid command on your array (or a partition under your array) to get its UUID, then use that UUID to mount it in /etc/fstab.

Example:

root@locutus:~# blkid /dev/mapper/vg0-root
/dev/mapper/vg0-root: UUID="61998221-7b39-49cd-83f7-62fda973218c" TYPE="ext4" 

With this information, I would set my /etc/fstab entry for root to look like this:

UUID=61998221-7b39-49cd-83f7-62fda973218c    /    ext4    defaults,errors=remount-ro    0    1

Once you've done this, your system will find your partitions no matter whether their device names change or not (for example, if drive cables get swapped around).

You should also set up your /etc/mdadm/mdadm.conf to find things automatically without needing any particular drive to be in any particular location:

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md/0 metadata=1.2 UUID=c913486a:e62c7ea1:cfb98b6b:253d1f62 name=lucas.mgscreativa.com.ar:0

NOTE THAT THERE ARE NO DEVICES SPECIFIED IN THIS CONF FILE. This is a feature, not a bug. With mdadm.conf set up this way, your system will automatically scan all attached drives looking for anything that belongs to your array, and will assemble them all as /dev/md0.

Solution 2

For this answer I am assuming that the array is intact - no failed disks, not other error conditions, just the changing "device number".

What should normally work is to stop the array:

# mdadm --verbose --misc --stop /dev/md127

... fix the line referring to the array in /etc/mdadm/mdadm.conf (i.e. adjust the name of the device). Then assemble it as follows:

# mdadm --verbose --assemble --update=super-minor --run /dev/md0 /dev/sdaX /dev/sdbX

(adjust the references to the physical devices to your case).

You should see an output like this:

mdadm: looking for devices for /dev/md0
mdadm: updating superblock of /dev/sdaX with minor number 0
mdadm: /dev/sdaX is identified as a member of /dev/md0, slot 0.
mdadm: updating superblock of /dev/sdbX with minor number 0
mdadm: /dev/sdbX is identified as a member of /dev/md0, slot 1.
mdadm: added /dev/sdbX to /dev/md0 as 1
mdadm: added /dev/sdaX to /dev/md0 as 0
mdadm: /dev/md0 has been started with 2 drives.

And if you query the details you should see an updated "Preferred Minor" field. Check with:

# mdadm -Q -D /dev/md0|grep "Preferred"
Preferred Minor : 0

Caveats: also, make sure that you mount the device and write to it. Otherwise the change may not take effect. Also, super-minor appears to be specific to the old metadata format.

NB: I do not know whether you can affect the "name" in the original sense with this. I know you can affect the X in the created /dev/mdX names, however.

Comments

[razor7]

Well, allways checked mdadm status and it was allways clean!, really doin't know why changed! Persistence : Superblock is persistent State : clean. Will Try with the UUID aproach, I knew that, but tryed with my array and didm't work, will give a second shot. Thanks!.

[razor7]

Well, I need some help here, cant fogure out the UUID mess BLKID:/dev/md127: UUID="fd719a38-58ca-4f3d-a98a-d04d703d1020" TYPE="ext4" /dev/sda1: UUID="d0dfd2f7-636c-4240-9609-0a0bce180a61" TYPE="ext4" /dev/sdb1: UUID="c06ffc6d-b146-4bcb-9d91-ac18507fc3bb" TYPE="ext4" /dev/sdc1: UUID="e1824d3c-6ea5-46b2-9e62-baacca4194c7" TYPE="ext2" /dev/sdc5: UUID="zXeL5s-2SVr-BciK-Bza5-iVnV-Q1iw-Hq2AbF" TYPE="LVM2_member" /dev/mapper/lucas-root: UUID="2acf0c5c-edc9-41f2-8858-8a1eff9e55fc" TYPE="ext4" /dev/mapper/lucas-swap_1: UUID="1d51b5fb-84c5-419d-b715-3b307551780a" TYPE="swap"

[razor7]

and mdadm --detail --scan ARRAY /dev/md/lucas.mgscreativa.com.ar:0 metadata=1.2 name=lucas.mgscreativa.com.ar:0 UUID=c913486a:e62c7ea1:cfb98b6b:253d1f62

[Jim Salter]

the mdadm.conf works as-is, the way I posted it above - it already has the correct UUID for your array in it. for fstab, do cat /etc/fstab and then do sudo blkid of each of the individual devices as mounted. replace the /dev/whatever bit at the beginning of the line in fstab with UUID=whaargarble where whaargarble is the output of sudo blkid /dev/whatever for that device; ie replace /dev/mapper/lucas-root with UUID=2acf0c5c-edc9-41f2-8858-8a1eff9e55fc.

[razor7]

Well ran blkid /dev/md127 and used that UUID to mdadm.conf and fstab, but my issue is that sometimes the mdXXX suddenly changes. Any idea why? Currently my device is started at boot, but sometimes not because of the mdXXX changes number. This hasn't happened about two months ago. Just for the record, I have an IDE drive for the system and a 2x2TB SATA RAID1 array with mdadm for the data.

[Jim Salter]

again, at this point it should not even matter if the device number changes - because you aren't using it for anything anymore, just the UUID, which is guaranteed to never change. I suspect you were having problems before because you had specified device names for the member drives in mdadm.conf, and if (for example) your BIOS decided to swap them, you'd end up with an arbitrarily named "other" array instead of your normal array.

[razor7]

Hi, wiil this affect my array contents?, I want to prevent any data loss! Thanks in advise!

[0xC0000022L]

@razor7: no, these contents won't affect the contents, but they will affect the superblock data and how the device appears to the system.

[razor7]

OK, if you say that any data loss will occur, will try thoose commands!

[razor7]

Well ran this command # mdadm --verbose --assemble --update=super-minor --run /dev/md0 /dev/sda /dev/sdb but gos this error message! mdadm: looking for devices for /dev/md0 mdadm: --update=super-minor not understood for 1.x metadata

[0xC0000022L]

@razor7: this is what I was saying with Also, super-minor appears to be specific to the old metadata format. in my answer. In this I case I don't know the answer, yet. But I'll have a look on one of the machines on Monday.

[Sidias-Korrado]

I know this is an old question and old answer, just an update: 1.2 superblock does not have an Preferred Minor.