BTRFS Raid 1 vs. MDADM Raid 1

Question

I'm currently in the process of setting up a NAS system mainly for backing up my computer and probably also for streaming media files to my devices.

For these purposes, I decided to use OpenMediaVault with two new 4GB drives as it seems that it's a stable running software and that it has all the needed features already built in.

But it's not very clear yet which filesystem to choose. I'm currently tending to go with BTRFS instead of ext4 as I really want to have the "self-healing" as well as simple snapshots.

Apart from that, I would like to set up my drives in RAID 1 so I can benefit from a higher data availability.

Now when it comes to BTRFS, there are two options on how to set up my RAID 1.

Either one is using the standard Linux software raid with mdadm and the other one would be "native" btrfs raid.

What are the advantages/disadvantages of each?

Solution 1

The main tradeoff is pretty simple to understand:

• BTRFS has better data safety, because the checksumming lets it ID which copy of a block is wrong when only one is wrong, and means it can tell if both copies are bad.
• MD RAID has better performance, because it does a better job of parallelizing writes and striping reads.

Realistically, raid1 mode is the most well tested of the various multi-device configurations provided by BTRFS, by virtue of the facts that it's one of the oldest supported configurations, and it's also one of the most widely used. Most of the developers consider it stable enough (if you're not using complicated features) to be a realistic option in production usage, provided of course that you stay up to date.

Solution 2

The current status of btrfs can be found on the btrfs wiki's “Status” article. btrfs RAID1 has been stable for a while, and has several useful features that mdraid does not (e.g., mirror is per-file, so you can have unmirrored files too if you want—though tools to do this aren't readily available). Also, adding a third drive (when you need more space) is much easier with btrfs than mdraid.

mdraid's main advantage is it's far more mature. So mdraid+ext4 is probably a little bit safer. Documentation & experts are easier to find. But if you're running btrfs on top of it anyway (because, e.g., you want snapshots)….

Personally, I have a set of external drives I use for one layer of backups; they use btrfs. I started with one drive (back in 2014), then later added a second (larger) drive for redundancy. Later the first drive failed, so I replaced it with another larger drive. Then I was going to run out of space, so I added a third drive, twice as big as the other two. Sometime in the next year or so, I'll have to add another. All of this would be much more painful (or at least time-consuming) to do with mdraid than it was with btrfs RAID1.

So, for your application, I'd suggest btrfs RAID1 is a better choice. If you were running a mission-critical darn-near-100% uptime required server, it'd be different. And of course, as always, keep backups (of your media files, at least if you care if they're lost — you don't want to know how many copies of my music collection I have...)

Solution 3

One advantage of mdadm raid is that it is well understood and stable. Also grub was previously not able to boot from multi device btrfs filesystems. There may be other software bugs with such setups. You also could use mdadm to create a raid10 instead of a raid1. Using the offset layout with a largeish chunk size gets you much of the performance of a raid0 with the redundancy of a raid1.

One advantage of the native btrfs method is that you can decide to convert from raid1 to raid0 later. Another theoretical advantage is that you can mix and match and put some files in raid0 and some in raid1, but the last time I checked at least, the tools weren't up to that yet.

Comments

[psusi]

Close voters: this is not primarily opinion based.

[psusi]

The last I heard, the filesystem is capable of having different chunks use different raid levels, but there was no user space method of flagging different files to be stored in a different chunk type. Is there a command for doing that these days?

[derobert]

@psusi I feel I've seen an a way to do that somewhere, but can't find it right now ☹. I'm sure the drange filter could be used, but that'd be a PITA.

[Csaba Toth]

I used btrfs multi device volume as a main volume for a good while. What's funny is when the data safety has to come into play (something goes south, logical errors reported) and you'd need to "fsck" it (quoting fsck intentionally, since btrfs has different mechanism), it's incredibly painful PITA. You'd think to somehow mount it in a recovery mode and then "fsck", it's almost impossible.

[Csaba Toth]

It's indeed possible to boot onto a multi-device "RAID0" btrfs volume, this is back from 2016 back askubuntu.com/questions/794763/…, I got that working as the only and main volume. But the installer was not able to understand the situation, failed, needed black magic and tricks to go through. Now 2 generations later I won't use any RAID, because I'm not sure about the speed benefits (phoronix.com/…) and it just complicates everything.

[TecHunter]

RHEL 8 and Centos 8 dropped BTRFS support. hit this wall so... fyi

[Austin Hemmelgarn]

@TecHunter And most of that was them deciding they didn’t want to support people who had no idea what they were doing. You shouldn’t be using any software for production workloads if you don’t understand it’s limitations, BTRFS is just a bit less forgiving in that respect than most other options.

[TecHunter]

@AustinHemmelgarn yes, it motivated me to move to MD Raid.

[Adam]

Well, BTRFS has checksumming and MD RAID does not. MD RAID does NOT guaranty data integrity at all. If a one disk in a MD mirror contains/returns corrupted data, this goes unnoticed and you cannot tell wich side of the mirror is the correct one! BTRFS would detect and fix this automatically. You can even scrub it. And BTRFS does support stripping. Due to its cow nature it is always consistent and snapshots are very efficient. From a data integrity and safety point of view, BTRFS is superior to MD+ext4.

[Adam]

And MD RAID is pretty brain dead. It does not know anything about the filesystem and which blocks actually contain data. So, rebuilding an array takes ages (with TB sized drives!) even if it actually way empty! IMHO BTRFS is the system of choice. If you are scared use ZFS, but BTRFS is much flexible, you can remove drives and change the raid level online. bit.ly/3pRAkGd