How to create an external selfhealing btrfs USB HDD RAID 1 array?

Question

The hardware is an external 2TB bus powered USB disk.

What I want is a RAID1 setup of 2x 1TB btrfs.

Is this possible? How?

__

Motivation

The motivation for me to use RAID1 btrfs is because I just read an article talking about how such a setup would repair corrupted files automatically when the checksum on one volume fails.

Example of what can happen with a .jpeg by just flipping one bit (I used vim for this. :%!xxd -b)

Solution 1

I assume you use 2 external drives or 1 enclosure housing 2 or more drives. Theoretically it shouldn't matter if the devices are connected via USB or SATA, btrfs should be able to assemble the array once it finds the individual drives/partitions.

You should have a look at the btrfs kernel wiki. The first usecase seems to cover what you asked for:

mkfs.btrfs -m raid1 -d raid1 /dev/sda1 /dev/sdb1

But I'm not sure about the selfhealing part. I only know that from ZFS.

Solution 2

This answer is a guide for showing how I got this going.

My setup is:

• 1 external USB drive (1 enclosure with 1 drive)

How to:

1. I created a GPT parition table with two equal sized primary btrfs partitions using gparted. Name the partitions whatever you want, the names will get lost anyway when creating the RAID.

   

2. Get the device ids:

   $ sudo btrfs filesystem show
   Label: none  uuid: 607b4153-7aa9-444d-bc15-c5fe9038f255
       Total devices 2 FS bytes used 28.00KB
       devid    1 size 1.82TB used 2.03GB path /dev/sdc
       *** Some devices missing
   Label: 'Max'  uuid: b6647427-9f27-4157-b47b-77f74054b885
       Total devices 1 FS bytes used 28.00KB
       devid    1 size 931.49GB used 2.04GB path /dev/sdc1        // first one
   Label: 'Moritz'  uuid: d0eaf97d-249e-4b7c-88a5-b60cc2d489d9
       Total devices 1 FS bytes used 28.00KB
       devid    1 size 931.49GB used 2.04GB path /dev/sdc2        // second one
   

3. Create the RAID:

   $ sudo mkfs.btrfs -L RAID-Datensicherung -m raid1 -d raid1 /dev/sdc1 /dev/sdc2
   

4. Since you can't write to it yet, you need to become the owner of the new device and change the permissions. Mount the device in nautiilus or however and then:

   $ cd /media
   $ sudo chown julien:julien RAID-Datensicherung
   $ sudo chmod 700 RAID-Datensicherung
   

5. Once you unmount and remount the device again, you can copy files via drag-and-drop.

When mounting the device in nautilus two volumes show up, but only one of them will show as mounted.

Once you copy files to the disk from another external drive you'll see that write speed to the RAID volume is twice than the read speed from the other external device because of the raid mirroring.

I still have to find out how to check if the self-healing works and will update this answer accordingly.

Update 1

User @Oli is asking the related question about flipping just one bit. Just like him, I need to do this stealthy, i.e. no timestamps of my interaction or anything like that.

If you change one bit while the file-system is mounted, the change is registered and immediately mirrored. The changed/'corrupted' file is seen as a mere update - not as bitrot.

Solution 3

About simulating a BitRot, why no one tell it can be easy done with dd linux command on an off-line as well as on an on-line BTRFS?

https://man7.org/linux/man-pages/man1/dd.1.html

Let me explain how:

1. Look for what devices are used by the BTRFS, let say they are /dev/sda1 and /dev/sdb1, just as an example.
2. Just run a dd command to overwrite a small portion of one of the devices

Example, overwrite only one byte with random data at position N (in bytes):

dd bs=1 count=1 if=/dev/random of=/dev/sda1 seek=N

Explained parameters:

• seek=N indicates where to write from start of the device (replace N with position in bytes units
• bs=1 indicates that units are in multiple of one byte
• count=1 indicates that only one chunk (of bs size) will be written
• if=... indicates where to read the data, (if ... is /dev/random it will be random data)
• of=/dev/sda1 indicates where to write data (replace it with the device path, not the mounted path)

And that is it, you have written random data at a position in the device (out of the BTRFS chain control, aka, BitRot simulated).

Another thing is how to calulate that N to be inside a specific file... but since what is wanted is test Anti-BitRot of BTRFS the test can be done with a random N in a BTRFS with one huge big file on it (90% used, ~10% free).

To test if simulated BitRot get fixed you can create a MD5 sum before dd command and after dd command, they must match.

So:

1. Create a BTRFS raid1 of two devices (better if they are not huge in size, just to test Anti-BitRot of BTRFS)
2. Write a file with random data that has a sice of arrounf 90% of total BTRF volume size
3. Create the MD5 sum of the file
4. Do the dd command
5. Create the MD5 sum of the file
6. Compare both MD5 sum, they must match

No matter if BTRFS is online or offline (aka, mounted or not mounted) when doing the dd command, since it is been done on one of the devices, not on the mount point of the BTRFS, the BTRFS will not know nothing about that write (BitRot simulated).

Hope this helps.

Comments

[king_julien]

Thanks, I'll look into this and share any news if I succeed.

[king_julien]

I've tried to get this working like ten times already in different ways. But I never managed to get write access to the raid volume and got weird error messages when trying to get there. Anyway, thanks for the link.

[king_julien]

I think I got it working after all: 1) sudo btrfs filesystem show to get the device names 2) sudo mkfs.btrfs -m raid1 -d raid1 /dev/sdd1 /dev/sdd2 3) Goto /media. A new volume has shown up. 4) sudo chown julien:julien 35b7e180-117c-4ecd-8852-d25c6cb724a8 5) sudo chmod 700 35b7e180-117c-4ecd-8852-d25c6cb724a8 6) Now unmount and remount the volume and you have write permissions in nautilus.

[king_julien]

When mounting the volume two disks show up, but if I click on either of them only the 'first' one is mounted. When I copy files to the disk, from another external drive I see that write speed to the RAID volume is twice than the read speed from the other external drive. This looks good. I still have to figure out how to rename the RAID volume and try to flip a bit to see if the auto-selfhealing works.

[Jim Salter]

Look at @Oli's question again. I answered about how to do an offline edit of a file to simulate on-disk corruption.

[DanMan]

This is not what the OP asked for, but about 2 actual drives.

[Csabi Vidó]

@DanMan Yes there was a little misunderstanding back then, that's why the OP published his own answer which got an upvote from me. What do you suggest for further improvement?

[DanMan]

@LiveWireBT I don't know. He should probably accept his own answer instead. I guess you can spare the few rep. points.

[basic6]

As for self-healing: How to monitor BTRFS filesystem raid for errors