How to modify/fix incorrectly detected dmraid (FakeRaid) RAID 10 array

Question

I'm trying to get dmraid to properly detect an existing RAID 10 array, which is working fine within windows 7. In summary, the drive setup and bios partitioning is as following:

• 2 x SSD (120 GB), RAID 0.

  Partitioned into: 180 GB (win7) and 58 GB (Ubuntu).

  Both detected and mounting fine in Ubuntu.

• 4 x HDD (2 TB), RAID 1+0. Single 4TB partition.

  Not working in ubuntu.

---

The Ubuntu installation itself is on a partitioned striped disk pair, and works fine. The two partitions (180 GB and 58 GB) correspond to the following LVM's (I've taken the liberty to format the data for readability):

#$ sudo dmsetup info /dev/dm-{1,2,3,4,5}
|----------+--------+---------------+-----------+----------+--------|
| dev/dm-0 | ubuntu | pdc_hjijcjji  | container |          | 58 GB  |
| dev/dm-1 | ubuntu | pdc_hjijcjji1 | /         | ext4     | 54 GB  |
| dev/dm-2 | ubuntu | pdc_hjijcjji2 |           | extended | 4.3 GB |
| dev/dm-3 | win 7  | pdc_fjhhdeeg  | container |          | 180 GB |
| dev/dm-4 | ubuntu | pdc_hjijcjji5 |           | swap     | 4.3 GB |
| dev/dm-5 | win 7  | pdc_fjhhdeeg1 |           | ntfs     | 180 GB |
|----------+--------+---------------+-----------+----------+--------|

The Raid 10 array (which is the one I need help to get it working within Ubuntu) consists of four 2TB disks, and give a resulting 4TB array. It seems as if dmraid is aware of this array, given the following output:

#$ sudo dmraid -r
|----------+--------+------------------+--------+---------+----------------+---------|
| Device   | Format | Name             | Type   | Status? | Size (sectors) | ?       |
|----------+--------+------------------+--------+---------+----------------+---------|
| /dev/sdf | pdc    | pdc_fjhhdeeg     | stripe | ok      |      175781248 | data@ 0 |
| /dev/sde | pdc    | pdc_fjhhdeeg     | stripe | ok      |      175781248 | data@ 0 |
| /dev/sdd | pdc    | pdc_bjibibahah-1 | stripe | ok      |     1758766336 | data@ 0 |
| /dev/sdc | pdc    | pdc_bjibibahah-1 | stripe | ok      |     1758766336 | data@ 0 |
| /dev/sda | pdc    | pdc_bjibibahah-0 | stripe | ok      |     1758766336 | data@ 0 |
| /dev/sdb | pdc    | pdc_bjibibahah-0 | stripe | ok      |     1758766336 | data@ 0 |
|----------+--------+------------------+--------+---------+----------------+---------|

Which throws me off a bit, since I'd expect the array, pdc_hjijcjji, to show up here as well. Perhaps, since it's a partition within a striped disk, it is included within pdc_fjhhdeeg. In any case, the striped array is running fine, so I'm not too worried about it.

pdc_hjijcjji shows up when running dmraid -s:

#$ sudo dmraid -s
|-----------+----------------+--------------+--------------|
| Name      | pdc_bjibibahah | pdc_fjhhdeeg | pdc_hjijcjji |
|-----------+----------------+--------------+--------------|
|           |       Superset |   Active Set |   Active Set |
| Size (-h) |       1.677 TB |     167.6 GB |      54.0 GB |
| Size      |     3517532672 |    351562496 |    113281024 |
| Stride    |            128 |          128 |          128 |
| Type      |         raid10 |       stripe |       stripe |
| Status    |             ok |           ok |           ok |
| Subsets   |              2 |            0 |            0 |
| Devs      |              4 |            2 |            2 |
| Spares    |              0 |            0 |            0 |
|-----------+----------------+--------------+--------------|
# Size is in blocks of 512 bytes.

Where pdc_bjibibahah seems to correspond to the Raid10 array, except that it has a weird size (1.677 TB, as apposed to approx 4 TB). Trying to activate this using dmraid -ay results in a mounted 1.677 TB device with is reported as unallocated data.

Running sudo dmraid -s -si on the bjibibahah array:

#$ sudo dmraid -s -si
|-----------+----------------+------------------+------------------|
| Name      | pdc_bjibibahah | pdc_bjibibahah-0 | pdc_bjibibahah-1 |
|-----------+----------------+------------------+------------------|
|           |       Superset |           Subset |           Subset |
| Size (-h) |       1.638 TB |         1.638 TB |         1.638 TB |
| Size      |     3517532672 |       3517532672 |       3517532672 |
| Stride    |            128 |              128 |              128 |
| Type      |         raid10 |           stripe |           stripe |
| Status    |             ok |               ok |               ok |
| Subsets   |              2 |                0 |                0 |
| Devs      |              4 |                2 |                2 |
| Spares    |              0 |                0 |                0 |
|-----------+----------------+------------------+------------------|

---

I've tried configuring the arrays manually, but with no success, and the man page isn't proving all too helpful either. If anyone has a suggestion as to how to configure dmraid, or to convince it to change the configuration of the RAID10 array, I would be really grateful.

Here are a few outputs, should it be relevant:

$ sudo dmraid -V
dmraid version:     1.0.0.rc16 (2009.09.16) shared
dmraid library version: 1.0.0.rc16 (2009.09.16)
device-mapper version:  4.20.0

$ sudo dmsetup --version
Library version:   1.02.48 (2010-05-20)
Driver version:    4.20.0

$ uname -srvm
Linux 3.0.0-16-generic #28-Ubuntu SMP Fri Jan 27 17:44:39 UTC 2012 x86_64

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 11.10
Release:    11.10
Codename:   oneiric

Also, the BIOS raid setup of the RAID10 array match the following disks within ubuntu: Code:

|---------+------------+--------------|
| Port:ID | Assignment | OS Disk Name |
|---------+------------+--------------|
|   01:01 | LD 1-1     | /dev/sda     |
|   02:01 | LD 1-2     | /dev/sdb     |
|   03:01 | LD 1-3     | /dev/sdc     |
|   04:01 | LD 1-4     | /dev/sdd     |
|---------+------------+--------------|

---

Update: I forgot to mention that I'm on a Crosshair V motherboard, using the built in AMD SB950 controller.

Update 2: When running sudo dmraid -s -ccs pdc I get the following error messages:

ERROR: pdc: wrong # of devices in RAID set "pdc_fjhhdeeg" [1/2] on /dev/sdf
ERROR: pdc: wrong # of devices in RAID set "pdc_fjhhdeeg" [1/2] on /dev/sde
ERROR: pdc: wrong # of devices in RAID set "pdc_bjibibahah-1" [1/2] on /dev/sdd
ERROR: pdc: wrong # of devices in RAID set "pdc_bjibibahah-1" [1/2] on /dev/sdc
ERROR: pdc: wrong # of devices in RAID set "pdc_bjibibahah-0" [1/2] on /dev/sdb
ERROR: pdc: wrong # of devices in RAID set "pdc_bjibibahah-0" [1/2] on /dev/sda
*ERR*

If I omit the specifier pdc, and just write sudo dmraid -s -ccs I get the even more cryptic:

*ERR*
*ERR*
*ERR*

Update 3: I've managed to remove the badly configured array using sudo dmraid -rE as suggested in an askubuntu QA here. I'm still getting the "ERROR: pdc: wrong # of devices in RAID set ..." errors, which seem to be impeding further progress. I'll keep searching for solutions to this problem.

Edit: Is there any information I can add that would help anyone help me?

Solution

OK :) So here's your goal:

4 x HDD (2 TB), RAID 1+0. Single 4TB partition.

Seeing that it's only reporting around 1.6TB, that much missing storage suggests a meta-data problem to me.

I would suggest tearing down the entire array and ensure the metadata is wiped by using dmraid -rE. Then, once those disks don't show up anymore, try making a simple RAID1 from the 2 2T disks, your container capacity should be closer to 1.8-1.9 then 1.6. Once that is verified, then create your stripe, which ought to just add the two together. If this doesn't work you might have a bug in either the firmware or the dmraid system itself.

dmraid is ummm... interesting, given the choice I would prefer not to use it. You're basically taking all the complexity of md-raid, without any increase in performance, and then mixing in a slew of different metadata formats from different vendors, and then sharing that metadata with multiple operating systems, trusting that the OS you shared it with doesn't ruin the metadata.

You might want to test actually replacing failed members after you get this configuration stable, a RAID is just a big single point of failure if you can't service it. Good luck.

In response to your dmraid -rE issue At this point you're only option is dd if=/dev/zero ... and wipe the disks out.

Comments

[Chuim]

I have almost exactly the same problem right now, even though mine is a raid-0 that should result in 4TB. Funny thing: the detected size from dmraid for both of is is precisely the same! I also have an AMD controller doing the raid thing and it's working perfectly on my Windows 7.

[Kevin Bowen]

possible duplicate: askubuntu.com/questions/877751/…

[Martina]

bugs.launchpad.net/ubuntu/+source/dmraid/+bug/599255