ZFS on top of iSCSI

Question

I'm planning on building out a file server using ZFS and BSD, and I was hoping to make it more expandable by attaching drives stored in other machines in the same rack via iSCSI (e.g., one machine is running ZFS, and others have iSCSI targets available to be connected to by the ZFS box and added to zpools).

Looking for other people who have tried this has pretty much lead me to resources about exposing iSCSI shares on top of ZFS, but nothing about the reverse. Primarily I have the following questions:

• Is iSCSI over gigabit ethernet fast enough for this purpose, or would I have to switch to 10GbE to get decent performance?
• What would happen when one of the machines running iSCSI targets disconnects from the network?
• Is there a better way to do this that I just am not clever enough to have realized?

Capacity wise, initially this would be about 10TB of data (not accounting for redundancy), and a reasonable goal for the foreseeable future would be scaling to 20TB, so with redundancy, probably about 40-50TB of total storage. If possible, we also want to keep all the data encrypted using GELI.

Thanks for any help.

Solution 1

What is your goal in terms of capacity? This is definitely possible, as ZFS would be able to see your targets and aggregate them into pools. However, you're leaving a good amount of performance and reliability on the table.

My recommendation for scaling (at what I assume to be your capacity needs) is to invest in an external SAS multipath-cabale drive enclosure and ZFS-friendly controllers. If this is a situation where you need more than say, the 24TB of usable RAID 1+0 storage provided by one external enclosure filled with 2TB disks, then you're in the realm where you would benefit from expert design advice. At that level, iSCSI using disks present in other systems won't make things any faster.

The iSCSI design will fall over for latency, reliability and supportability reasons.

Solution 2

As for your question about iSCSI over ethernet - iSCSI is just about the fastest protocol you'll find. It's basically straight disk block access over a wire. It will saturate a decent gigabit NIC.

Concerning losing targets on the network, (almost) every iSCSI target implementation that I've seen supports multipath of some sorts. I think open-ietd might not yet support multipathing. At worst, you're using a journaling file system; you might have to replay the journal when the target comes back online. I've yet to corrupt a file system over iSCSI from dropping a storage server.

Solution 3

Note: I haven't actually done this, so take this with a pinch of salt. I have seen mention of this while reading about ZFS, but can't find these references now...

You want to export each physical disk as a separate LUN so that ZFS has full visibility into the physical layout. This is necessary for it to make the right decisions about IO scheduling and replication.

Is iSCSI over gigabit ethernet fast enough for this purpose, or would I have to switch to 10GbE to get decent performance?

It depends on the speed of the disks, how many disks there are, and what performance you want to achieve. 15k RPM disks can transfer up to 105MiB/s, which is 840Mbit/s. Accessing more than one such disk over a single Gigabit link will saturate the link and make your network the bottleneck. Find the maximum speed of the disks you want to use, multiply by the number of disks, and you'll get the network bandwidth you need to support that.

This does of course assume that you want the maximum performance possible out of the ZFS server. If you only have a handful of clients connected over 100Mbit/s this isn't necessary, so calculate what the maximum demand you expect is. Bear in mind that the bandwidth to the disks is slightly higher than the client bandwidth if you're using RAIDZ1/2/3, and of course, if the server accesses the disks over the same NIC as the clients access the server, this bandwidth needs to be shared.

What would happen when one of the machines running iSCSI targets disconnects from the network?

ZFS would see the disk as becoming unavailable. If you're using RAID1/2/3 this shouldn't interrupt service to clients. If you've configured a hot spare ZFS will start resyncing data to it. When the iSCSI target comes back ZFS should start using it again, assuming that the initiator automatically reconnects. (This is something you should test though.)

Comments

[Tek]

Did you ever test this? How'd it turn out?

[Solipsism]

added some info to the question regarding capacity; in brief, we'd probably be safe if we were able to scale well up to about 20TB of redundant storage

[ewwhite]

Is there a cost or budget reason you couldn't do this with enclosures and disks, avoiding iSCSI?