How to upgrade a ZFS RAID-Z array to larger disks on OpenSolaris?

Question

I'm planning on installing OpenSolaris on my home server (right now it has Linux) and I would like to know how to prepare for upgrading the server to have larger hard drives in the future. Now the server has 4x 400GB SATA drives and I would install OpenSolaris with ZFS RAID-Z on them. Some time later this or next year I'm going to upgrade the drives to maybe 4x 1TB SATA drives.

How can I do the upgrading and increasing of the partition size with ZFS? Is it a simple and painless operation (e.g. replace one drive at a time and ZFS will automatically resize itself to use the full space), or are some special steps required? Can it be done without shutting down the system?

Solution 1

1. Offline drive #1.
2. Replace the old drive with the newer, high-capacity drive.
3. zpool replace the drive.
4. Wait for the replacement to be rebuilt.
5. Goto step 1 and repeat for each successive drive in the array.

You may need to export and then import the zpool once this procedure is complete; the pool will then show the increased size. No downtime is required, but if a drive does go south while you're upgrading your array, you will lose everything, as with four total disks you presumably have a 3+1 setup. (The odds aren't that high, but Murphy never sleeps.)

The official source: "Replacing Devices in a Storage Pool" in the ZFS Administration Guide. The ZFS Best Practices Guide is also useful.

Solution 2

I would like to add that replacing a non defective drive makes the raid less fail proof. You can actually fill the replacing drive first with data before taking out the smaller drive. So the order will become.

• Attach bigger drive (can even be through usb)
• zpool replace smallerdrive biggerdrive
• wait
• when it is done take out smaller drive, or whatever you need to do.
• repeat
• and at the end you might need to run zpool set autoexpand=on pool

Now murphy is less of a problem

Comments

[Josh]

+1 for "You may need to export and then import the zpool once this procedure is complete" -- the procedure of replacing wasn't working for me until I took this action.

[bahamat]

You don't need to reimport it if the autoexpand option is set to on for the pool.

[Peterdk]

For people doing this on Linux: I replaced each drive succesfully, but it would not increase space when running autoexpand=on. I needed to run zpool online -e poolname devicename for each device, and then it worked. Importing/exporting did nothing.

[Tony]

for other zfs newbies, a hint that the zpool replace command is zpool replace <poolname> <smallerdrive> <biggerdrive>. That is, it needs the pool name before the device names