Btrfs: RAID 1 on 3+ devices

Question

I currently have a Btrfs partition with four devices: three 3 TB drives and a 4 TB drive. Data and metadata are RAID 10, so I have 6 TB of usable space, which is almost full. I'd anticipated that RAID 5 support in Btrfs would be mature by the time my storage filled up, but apparently it's not a priority.

My question is: is there a reason to prefer RAID 10 over RAID 1? I know real RAID 1 on my current hardware should give me 3 TB of usable space with 4 copies of each block, but Btrfs apparently does not behave this way. From the Btrfs FAQ:

btrfs combines all the devices into a storage pool first, and then duplicates the chunks as file data is created. RAID-1 is defined currently as "2 copies of all the data on different devices". This differs from MD-RAID and dmraid, in that those make exactly n copies for n devices. In a btrfs RAID-1 on three 1 TB devices we get 1.5 TB of usable data. Because each block is only copied to 2 devices, writing a given block only requires exactly 2 devices to be written to; reading can be made from only one.

And from Jens Erat on Stack Overflow:

Btrfs distributes the data (and its RAID 1 copies) block-wise, thus deals very well with hard disks of different size. You will receive the sum of all hard disks, divided by two – and do not need to think on how to put them together in similar sized pairs.

If more than one disk fails, you're always in danger of losing data: RAID 1 cannot deal with losing two disks at the same time. In your example given above, if the wrong two disks die, you always lose data.

Does this mean that rebalancing from RAID 10 to RAID 1 will give me an extra 500 GB of data (6.5 TB rather than 6 TB) due to the 4 TB drive? And is there any reason for me to stick with RAID 10?

Solution

Yes, you get an extra 500 GB. Note that determining available space in btrfs remains elusive. Also: have a look at the btrfs disk usage calculator.

On your second question: You may lose some performance on your array. Naturally, your data is equally safe on both RAID configurations. When considering performance you can perhaps have a look at these benchmarks: kernel.org, phoronix.com.

Have you perhaps already tried converting to RAID 1? If so: what are your findings?

Comments

[basic6]

BTRFS RAID-5 is (although still experimental) actually pretty stable and even many typical data recovery scenarios work in the current version (without crashing). You should probably scrub from time to time to make sure your data is still correct, but it might be worth a shot. It's possible to convert from RAID-1(0) to RAID-5.

[Mikkel]

@basic6 Good to know, thanks. I was researching that not long ago and saw that scrub and replace support had been added as of 3.19, but people were still complaining about the lack of devices failure alerts. I do have crons set up for weekly scrubs and weekly/monthly SMART self tests, so I should be able to catch those issues. I'll give it a try.

[basic6]

Periodic (weekly or monthly) scrubs are important, if a scrub finds and error because it can't read from a drive, it will increase the error count. Check the error count using dev stats, which could be another cronjob.

[DavidPostill]

This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post - you can always comment on your own posts, and once you have sufficient reputation you will be able to comment on any post.

[Mikkel]

However, to answer your question, I did convert and the result seems to be consistent with my expectations. Btrfs does a poor job of indicating actual available disk space, but you can see in the output of btrfs fi sh that the extra space on the 4 TB drive is being taken advantage of, and (2.83+1.93+1.93+1.95)/2~=4.30 as expected. I recently sustained a drive failure and successfully rebuilt with no data loss, so I can confirm firsthand that redundancy is intact.

[Laura]

I'm sorry my reply was unclear @david. I attempted to answer the two questions in the last two lines of the original post. Additionally I indicated a range of uncertainty (which seems intrinsic to btrfs) and supplied my sources for reference. Indeed I also asked a question to the author, so others could perhaps learn from his findings. I did this within the answer for reasons you already explained. I will consider your comments next time I answer a question on stack exchange.

[Mikkel]

It may not be the most certain answer, but my results indicate that your assumptions are correct, so I'm accepting it. I'm not certain about your comment on performance differences, though. It's certainly true of hardware RAID 1 vs 10, but I'm not sure if Btrfs sees a performance boost from RAID 10. Furthermore, what it calls RAID 1 is still striped across multiple devices here, so it's still more akin to RAID 10 than real RAID 1. If I'd read your reply before converting, I would've done before and after benchmarks, but I'm not spending another week converting the array to RAID 10 and back.

[Mikkel]

Well, there is this page, which shows marginal benefits to RAID 10 over RAID 1, carrying the caveat at the top that the benchmarks are 7 years old.