Consumer (or prosumer) SSD's vs. fast HDD in a server environment

Question

What are the pro's and con's of consumer SSDs vs. fast 10-15k spinning drives in a server environment? We cannot use enterprise SSDs in our case as they are prohibitively expensive. Here's some notes about our particular use case:

• Hypervisor with 5-10 VM's max. No individual VM will be crazy i/o intensive.
• Internal RAID 10, no SAN/NAS...

I know that enterprise SSDs:

1. are rated for longer lifespans
2. and perform more consistently over long periods

than consumer SSDs... but does that mean consumer SSDs are completely unsuitable for a server environment, or will they still perform better than fast spinning drives?

Since we're protected via RAID/backup, I'm more concerned about performance over lifespan (as long as lifespan isn't expected to be crazy low).

Solution 1

Note: This answer is specific to the server components described in the OP's comment.

• Compatibility is going to dictate everything here.
• Dell PERC array controllers are LSI devices. So anything that works on an LSI controller should be okay.
• Your ability to monitor the health of your RAID array is paramount. Since this is Dell, ensure you have the appropriate agents, alarms and monitoring in place to report on errors from your PERC controller.
• Don't use RAID5. We don't do that anymore in the sysadmin world.
• Keep a cold spare handy.
• You don't necessarily have to go to a consumer disk. There are enterprise SSD drives available at all price points. I urge people to buy SAS SSDs instead of SATA wherever possible.
• In addition, you can probably find better pricing on the officially supported equipment as well (nobody pays retail).
• Don't listen to voodoo about rotating SSD drives out to try to outsmart the RAID controller or its wear-leveling algorithms. The use case you've described won't have a significant impact on the life of the disks.

Also see: Are SSD drives as reliable as mechanical drives (2013)?

Solution 2

Yes, the SSDs will be way faster than the SAS drives. For sequential throughput, a good RAID of SAS drives might do pretty well, but for random access, the SSDs will blow them out of the water which can result in a very noticeable performance difference.

Depending on the particular SAS drives and the particular SSD drives, the SSDs may have a better unrecoverable read error rate by up to a factor of 10.

Some tips for if you do use consumer SSD drives:

• Know your write workload so you can estimate how often you'll have to replace the drives since they have a certain amount of write endurance
• If you can spare the space, overprovision the drives to make them more like enterprise ones
• Check out articles comparing the performance and write endurance characteristics of SSDs in the the same class and pick the one best suited to your needs
• Personally I'd get SSDs with a 5 year warranty because I believe the manufacturer is going to provide better quality as a result. I know this isn't a hard and fast rule, just personal belief.
• There are low end consumer SSD drives and higher end one - sometimes labeled something like "Pro" - you might want to look for ones in that class
• This goes for enterprise drives too, but be sure you're monitoring the MWI (media wear indicator) so you know when to replace the drives

Solution 3

Even consumer-grade SSDs are much faster than the faster 15k HDDs, so from a performance standpoint they will be fine (if using the right disk and if overprovisioning them), but you had to carefully pick them, especially due to how they interact with hardware-based RAID controller...

1. First, check if affordable, entry-level enterprise grade drive (as Intel S3500/S3600, Micron M500DC and Micron M510DC) are within your reach. If so, you can skip the whole consumer-grade lottery.
2. Check whether your RAID cards support 3rd party disks. For example, earlier DELL firmware for H700/H710/H710p cards refused to initialize non DELL-rebranded disks. A subsequent update initialized such disks, but marked the array "degraded". Only relatively recent (end 2013) firmware updates corrected that precarious situation.
3. Keep your disk's private cache enabled. Some RAID card will forcibidy disable the disk's private cache. This kill performance for consumer-level SSD, as they make heavy use of private DRAM cache both to cache their indirection table and to mask the heavy latency involved into erasing/programming MLC NAND. For example, an otherwise very fast Crucial M550 240GB drive write at incredibly slow rate of 5 MB/S when its internal cache is disabled.
4. If possible, strongly favor disks with FULL power-loss protection. This put squarely in the enterprise champ but, as stated above, there are relatively cheap disks in this champ.
5. If no full power-loss protected SSD are in your shop list, at least use disk with partial power-loss for data-at-rest protection. Some excellent driver with such protection are the Crucial/Micron M500/M550 and the newer M600. Micron even has an interesting document on how/why overprovision its M600 drive for use in virtualization environment. Anyway, remember that with non-full power loss protected drives, it remain a small possibility to lose/corrupt your data. How small? it depends on your RAID controller behavior (for example, if it issues a final ATA flush command after transferring data to a cache-enabled disk) and on the disk's firmware, so it is not possible to give you a detailed answer. What I can say is that on all my tests, PERC RAID cards seems to always flush the disk's private cache (if it is enabled)
6. Strongly over-provision your consumer drives, at least with a 25/30 % reserved capacity.
7. Do not use second-class consumer drives. Even good consumer drivers have their problems, and going with a lower-tier consumer disk is asking for troubles.

Solution 4

Consumer grade SSDs will work fine in many servers for use cases.

They are way, way faster then SAS disks. I'd suggest the reason to get enterprise disks over consumer disks is not the speed, its the read-write cycles and better engineering - for example supercaps are present in some enterprise SSD's where the consumer grade version does not have this - if you loose power to the server your data is less likely to be killed.

You need to be aware that RAID is not backup - if you are going to RAID a couple of SSD's thats fine, but get different brands of SSD's, or at least different models so they have different performance characteristics. WHEN SSD'S DIE THEY ARE WAY MORE LIKELY TO DO SO WITHOUT WARNING, AND NO ABILITY TO PULL DATA OFF - on the flip side they are 10x as reliable as regular hard disks.

Look into the Samsung 850 series disks - at least for 1/2 the array - they are/were prosumer and offer good bang for buck, and are touted as being more reliable then 2d nand. (They use 3d nand).

Also, as someone else mentioned, don't do RAID5. Drives hold to much for it to work reliably - and back up your data.

Solution 5

If you are using them for writes, to avoid data corruption in the event of power failure you need to make sure that you only consider models with a supercap. Eg. Intel S3500, Samsung 845DC Pro

Otherwise consumer SSDs are more suited to caching.

View more solutions

Comments

[ewwhite]

Please provide specifics on the makes/models of hardware involved. And operating systems... and hypervisors... Maybe even what the VMs will be doing. More details!!

[David Budiac]

@ewwhite Dell rack servers. Likely an R430 or R730 with a PERC H730 RAID controller. Also likely a HyperV server hosting mostly Windows Server Standard... might use VMware over HyperV. Still considering. Initially VMs will be: domain controller, DNS, WSUS, deployment services. May add internal web server as well.

[ewwhite]

And how much capacity do you require?

[David Budiac]

@ewwhite 2TB usable at minimum

[hey]

superuser.com/questions/834521/… worth a read. Its about 10K consumer drives but many of the points are still relevant here.

[Greg]

We had issues with a Dell server and Samsung Pro SSDs - the RAID controller did not recognize them

[David Budiac]

@Gugges Yikes. Though in the end, the article mentions that it was a bug in the Linux kernel, not the SSD's

[Arthur Kay]

Consider storage spaces on Windows, zfs on Linux. Performance is good compared to hardware raid and very easily recoverable from using different hardware.

[David Budiac]

I've casually heard this before... not to use RAID5 anymore. Mainly because of reliability? And what do you use in it's place RAID6? RAID10?

[ewwhite]

RAID 1+0. See: Dell "RAID 5 is no longer recommended for any business critical information on any drive type

[ewwhite]

Overprovisioning is a good tip for the heavy write workloads.

[hey]

Just to add, The 850 pro is the one to go for. the standard 850 uses TLC

[davidgo]

@JourneymanGeek - I think the 850 EVO and 850 Pro both use 3d nand - it is the 840 series that doesn't. This is backed up by Samsungs site - samsung.com/global/business/semiconductor/minisite/SSD/globa‌​l/… - I'm a lot less certain, but I think the 850 pro has supercaps and better engineering, but the memory is very similar if not identical.

[user253751]

@ewwhite One URE every 200M sectors? If you do a full format, that would be on average >1 URE right out of the box of a 1TB drive!

[Martin Beier]

@davidgo That's why you buy Intel ;). Unless things have changed, the Intel SSDs will stop accepting writes when they fail and remain readable so that data can be copied off. techreport.com/review/26523/…

[s1lv3r]

+1 Good answer. I really wouldn't consider buying consumer grade SSDs for use on a PERC H700/H710/H730. Just Google for "PERC H730 uncertified drives" a lot people have tried that before and ended up with problems. Atleast buy cheap entry level SSDs like the already mentioned Intel S3500.

[Sebb]

Good answer, but it would be nice if you would add some reasoning behind your suggestions (eg. that link you posted in the comments). Why compatibility is important is obvious, but why are you favoring SAS over SATA?

[ewwhite]

@Sebb Already wrote about it.

[Martin Beier]

@MSalters Its not supposed to.... (it is supposed to remain readable - just not writeable)

[David Budiac]

@s1lv3r good point. Several sources mention that while Dell allows 3rd party drives, they're not certified and OpenManage will still give warnings. Though I can't seem to find a list of certified SSD's supported by Dell or the H730 :/

[davidgo]

I upvoted this even though I disagree. Certainly supercaps etc are a good idea for an SSD - hence the upvote - but it implies that consumer SSD's are unreliable - I don't believe this is correct - in fact I assert they are 10 times as reliable as spinning hard drives. Also, hard drives don't have supercaps or equivalent - and indeed are more vulnerable to power outages. The thing is that modern filesystems have journals to mitigate the risk of loss (and there are certain speedups you should not use on a drive unless it has a supercap / battery backup)

[JamesRyan]

@davidgo because HDs don't lie about caching sync writes whereas a lot of consumer SSDs do

[s1lv3r]

@DavidBudiac I don't think you'll find such a list - if it would exist, nobody would buy vendor supplied disks. ;-) Best I could find for you - some random guy on a forum, stating the S3500 works for him: "I have confirmed that the Intel S3500 works with the PERC H730 ...". ... - also as awwhite suggested, the card is LSI rebranded - IMHO it's an MegaRAID 9361 (LSI 3108) - so everything that works there should work with the PERC (the cards can even be cross-flashed if you are feeling brave enough).

[hey]

@davidgo non pros are TLC (and the 850s use 3d), EVOs iirc use ram to cache a bit more agressively and may have some slc or mlc cache internally for better performance. Pro is MLC.

[Joel Coel]

This, this, 1000 time this. Comsumer SSDs will lie to the RAID controller, making data loss possible even in highly-redundant RAID designs. If you use SSDs in the enterprise, you want the onboard capacitor.

[Joel Coel]

Enterprise SSDs have an onboard capacitor to guard against sudden power loss. Consumer SSDs not only lack this, but will also lie to RAID controllers about having correctly flushed volatile buffers, making them vulnerable to data loss even in highly-redundant RAID configurations.

[David Budiac]

FWIW, I spoke with a Dell rep. While they don't have an official list of SSDs that are compatible w/ the H730, he did say that OpenManage will not complain about any of the drives the Dell sells on their website. I just bought a handful of Samsung 845DC Evo's. Will report back w/ their compatibility.

[Dan Pritts]

regarding SAS vs. SATA: Are you really saying you recommend enterprise SSDs (regardless of connectivity) over consumer ones? Or do you really think SAS has a critical feature?

[ewwhite]

I recommend SAS or PCIe SSDs when I can. I just don't like SATA because it doesn't fit my use cases.