How many drives do I need for ZFS RAID-Z2?

Question

How much storage would I need if I wanted to use RAID-Z2? I am building a home server that will run FreeNAS with a ZFS file system, and currently plan to install 3 3TB hard drives.

• Does RAID-Z2 work on this?
• Would I need to downgrade to RAID-Z1?
• How many drives are needed for RAID-Z2?

Solution

RAID-Z2 works by taking the drives given to the vdev, and using two of those to store redundant data for recovery purposes in case something goes wrong.

Hence, the absolute minimum number of drives in a RAID-Z2 vdev is the two redundant ones plus one, which works out to three drives.

However, by doing it that way, you:

• are getting the effective storage capacity of only one of the drives
• can lose any two of the three drives involved without data loss (in the general case, you can survive the loss of any 2 drives in a N drive RAID-Z2 vdev)
• are forcing the system to calculate parity data for RAID-Z2

If you want a vdev consiting of three drives, or if you have three and want two drives' worth of redundancy, you are probably better off by setting them up as a mirror instead. That way, you:

• are getting the effective storage capacity of only one of the drives (regardless of the number of mirror devices)
• can survive the loss of any two of the three drives involved without data loss (in the general case, you can survive the loss of any N-1 drives in a N drive mirror vdev)
• are avoiding parity data calculations, because mirrors are bit-identical copies of each other; writing the same data to multiple disks is much cheaper than calculating separate parity data and writing that out to disk

The minimum number of drives where RAID-Z2 makes sense is four, which gives you the effective storage capacity of two of the drives and allows losing any two of the drives. This is advantageous over using the same four drives in a 2x2 mirror setup, because if you have two mirror vdevs of two drives each and lose some combinations of two drives, that vdev is dead and takes the pool with it in its fiery demise. In a 2x2 mirror vdev pool, at least one of the drives in each mirror pair must remain functional for the pool to remain available. However, the mirror may very well have better performance. As always, it's a trade-off between different choices, and one part of the system administrator's job description is to make those trade-offs appropriately for each specific situation.

Thus, while RAID-Z2 might technically work with the setup you describe, it offers no advantages whatsoever and even has some disadvantages compared to a simple three-way-mirror configuration.

It would be different if ZFS allowed us to dynamically change the redundancy level of or increase the number of devices in a RAID-Zn vdev, but it does not. The only way to actually grow a pool is by adding more vdevs or grow the size of the existing devices, not by adding devices to an existing vdev. (You can add and remove devices in a mirror vdev, but that only changes the amount of redundancy, not the usable storage capacity.)

You could also set up the three drives as a RAID-Z(1) vdev, which will give you the ability to survive the loss of any one of the drives before your data is at risk if anything whatsoever further goes wrong, reduce the CPU workload as compared to RAID-Z2 (because RAID-Z1 parity calculations are less computationally intensive) and give you the effective storage capacity of two of the drives combined.

Always keep in mind that if anything goes wrong, a full ZFS resilver is an arduous process for the remaining disks, and it isn't unheard of for another disk to develop problems or even die under the stress. For this reason, especially with rotational drives of the sizes common today, double redundancy should be the default choice with triple redundancy an option if you are really paranoid. Single redundancy should be considered only for less-important data where downtime can be tolerated. Single redundancy may be acceptable with a SSD-backed pool, however.

Hence my recommendation: If you want three drives ZFS, and want redundancy, set them up as a three-way mirror vdev. If you want RAID-Z2, use a minimum of four drives, but keep in mind that you lock in the number of drives in the vdev at the time of vdev creation. Currently, the only way to grow a ZFS pool is by adding additional vdevs, or increasing the size of the devices making up a vdev, or creating a new pool and transferring the data. You cannot increase the pool's storage capacity by adding devices to an existing vdev.

You also need to make sure you keep backups. This is doubly important with ZFS, because ZFS recovery after an actual failure that brings the pool below its redundancy threshold is very hard. ZFS has a very complex on-disk format that I am not aware of any common off-the-shelf data recovery software understanding, so even if you can read most of the data off the disks, if ZFS can't figure it out then you may very well be out of luck unless you are willing to spend tens of thousands of dollars on the problem. Backups are cheaper.

As an alternative to the three-way mirror, you could also run a two-way mirror with a spare, but most of the time, and certainly in the situation you describe, there is no advantage and some disadvantage to that.

As an aside, related to the issue of recovery, you should very strongly consider using ECC RAM in any system that runs a checksumming self-repairing file system including ZFS. Others agree.

Showing that RAID-Z2 with three devices is possible (this is ZFS On Linux 0.6.4). Notice that after hard removing two of the three backing files, "Sufficient replicas exist for the pool to continue functioning in a degraded state." and the vdev is DEGRADED, not FAULTED.

# truncate -s 1G /root/d1 /root/d2 /root/d3
# zpool create tank raidz2 /root/d1 /root/d2 /root/d3
# zpool status tank
  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        tank          ONLINE       0     0     0
          raidz2-0    ONLINE       0     0     0
            /root/d1  ONLINE       0     0     0
            /root/d2  ONLINE       0     0     0
            /root/d3  ONLINE       0     0     0

errors: No known data errors
# zpool export tank
# rm /root/d1 /root/d2
# zpool import tank -d /root
# zpool scrub tank
# zpool status tank
  pool: tank
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://zfsonlinux.org/msg/ZFS-8000-2Q
  scan: scrub repaired 0 in 0h0m with 0 errors on Tue Mar 29 11:00:23 2016
config:

        NAME                      STATE     READ WRITE CKSUM
        tank                      DEGRADED     0     0     0
          raidz2-0                DEGRADED     0     0     0
            18130982121682915530  UNAVAIL      0     0     0  was /root/d1
            18289483070703159278  UNAVAIL      0     0     0  was /root/d2
            /root/d3              ONLINE       0     0     0

errors: No known data errors
#