Using SATA III SSDs for ZFS pools

Question

I'm thinking of building a ZFS server optimized for random access (PostgreSQL database). I figured that I have 2 options:

1. Use pools consisting of the Crucial M500 960 GB SATA III SSDs. Enterprise SAS SSDs are not price-effective, considering that they could cost 10x as much as their SATA III counterparts.

2. Use pools consisting of 15K SAS HDDs. (In contrast, the price differential between an enterprise SAS HDD and a regular SATA III HDD is much smaller, so buying a SAS HDD is still justifiable.)

My questions are:

1. SATA III SSD vs 15k SAS HDD: Most of the arguments against SATA III have to do with the duty cycle (not suitable for 24/7?), but given that the MTBFs of consumer SATA III SSDs (e.g. Samsung 840 Pro, Crucial M500) are comparable to those of SAS SSDs and 15k HDDs, is there anything wrong with using the M500s? Or am I better off with the 15k SAS HDDs despite the clear performance advantages of the SSDs for random access?

2. L2ARC and SSD ZFS pools: If I'm using SSDs-only for my ZFS pools, would I benefit from an L2ARC device?

Solution

My recommendations vary a bit based on the distribution and approach to ZFS...

• SSD-only pools work well - However, you need to think very carefully about the controller, backplane and drive layout. Main considerations are: no expanders, use pure SAS HBAs, ZFS mirrors are king.

• 15k SAS disks in 2014 are Meh - If you need the random I/O performance of SSDs. You can used SAS disks as another tier or even nearline drives if you need the capacity.

• L2ARC won't be necessary for your all-SSD pool - Optimize and maximize RAM. There are some considerations for Linux-based ZFS solutions, but in general, more RAM is better than gobs of L2ARC, especially if you don't know your I/O pattern and working set.

• Miscellaneous - Use compression, monitor and scrub your pool, use good RAM, test, buy quality SSDs... Crucial isn't on my radar. There are other high quality SATA SSDs available that will have better performance characteristics.

Comments

[ewwhite]

Linux? OpenIndiana? SmartOS? OmniOS? FreeBSD? How will you be implementing ZFS?

[elleciel]

Probably ZFS on Linux... maybe Solaris.

[ewwhite]

Ooops, forgot Solaris ;)

[elleciel]

;) No problem...

[elleciel]

I've heard that the algo prioritizes most frequently-used data set for L2ARC and most recently-used data set for the ARC, so assuming my most recently and most frequently-used data sets are different, there's benefit to having an L2ARC cache even if I had a huge amount of RAM - is this correct?

[ewwhite]

@elleciel Depends on working set size and distro. My recommendation for Solaris-derivatives is different than Linux... For the former, I'd use L2ARC. For the latter... I've been doing some cool stuff with ZRAM and L2ARC. A lot of things need to be set in order for ZFS to behave on Linux, so I can only give a general answer now. Also, your SSD matters. For a high-quality low-latency SAS SSD, I would be okay going to the pool most of the time.

[elleciel]

Also, let me get your first point correct (backplane and drive layout). I should (1) have the disks placed internally, i.e. in the same chassis as the head node, instead of placing them behind an SAS backplane on an external JBOD chassis, and (2) connect my SATA drives directly to pure SAS controllers using SFF-8087-to-4x SATA cables instead of using SAS-SATA interposers?

[elleciel]

I see, that's helpful... I'll almost certainly be using SATA SSDs as SAS SSDs are too pricey.

[ewwhite]

@elleciel Depends on the chassis, the server backplane and the disks. If you were to say Supermicro was in use for JBODs, I'd say "stick with SAS". If you were to say an HP JBOD was in use, you could do SATA. In either case, you lose SAS multipath... so there are some design considerations there, too.

[ewwhite]

A best case is a server with 1:1 controller lane to slot mappings... So how many disks are you planning? How much capacity do you need and what hardware are you planning to use?

[elleciel]

Can't thank you enough. My original plan was 2 RAIDZ-3 zpools of 11 disks each. Each disk would be a 960 GB SATA SSD. We use Supermicro exclusively... Exact case and mobo would depend on the other design factors but I was tentatively planning to use a Supermicro 216BA-R920WB. I'm guessing this rules out the SATA SSDs and I'm forced by budget to go with the 15k SAS disks... :/

[ewwhite]

@elleciel you can use Supermicro... But look here for more best-practices: nex7.blogspot.com/2013/03/readme1st.html

[elleciel]

Gotcha, thanks for the link. I'll go back and do some homework based on your recommendations.

[ewwhite]

In general, RAIDZ3 in the config you're suggesting will be very slow.

[ewwhite]

@elleciel ZFS scales across vdevs. Your proposed design would have the random IOPS performance equivalent to two disks. Use mirrors when and if you can.

[elleciel]

Got it, I've a better idea how it works now. What about something like 6 RAIDZ-3 vdevs of 7 disks each? Trumps 14 3-way mirrors in capacity, better redundancy, still cheaper (capacity/$) than a hardware RAID-10 build.