Cheap but Highly Available Shared Storage?

Question

my group is in the market for plentiful storage. We traditionally used big-iron FC-attached SANs, but they are way expensive and provide much more performance than we require.

We want highly available shared storage, and if it performs like local desktop-class drives, that's fine.

Are there shared disk solutions out there that can get us slower but bigger disks?

We like NexSan SataBeast but they don't have a fully global presence which makes support awkward (we're a global company).

Is there any storage of this type that you would recommend? We'd prefer to use FC-attached storage but we're open to suggestions.

Thanks,

Wout.

EDIT: To answer the requests for clarification, we'd like

• globally available hardware support
• tens of TB with the ability to grow
• Highly Available: Ability to provide service through common failures, so:
  • hotpluggable drives
  • multipath IO
  • active/active controller
  • Anything like this that has a low cost/benefit ratio
• NFS only might be an option depending on HA-ness and performance
• Price: whatever it costs, but as little as possible

Basically I'm wondering what the 3D graph of price-performance-availability looks like currently. If there are sharp exponential curves, I'd be interested in the systems in those elbows.

As noted below, this could be done by a farm of PC-level hardware serving iSCSI and host-level software RAID6. This proves that whatever an EMC top-end storage array costs, it is probably too much.

Solution 1

Depends on your definition of "Cheap". But you could look at HP LeftHand Starter SANs. It's an all iSCSI solution but provides better performance the more you scale it, uses standard servers for the hardware, has solutions in SATA and SAS varieties to fit your budget and performance needs, and is supported by a global, good customer service, company. The boxes can be setup with no redundancy for maximum disk space (and minimal cost), or up to several layers of redundancy so your data basically cannot be lost short of an act of God.

The performance is comparable to FC, and even can even surpass FC the more you scale it.

Solution 2

How much storage do you need? If you want something in the 10's to 100's of Terabytes you could look at the Sun X4540 (Thumper). Solaris comes with an iSCSI target if you need block level storage or it can be used as a NAS with NFS or Samba.

For those not familiar with Sun's Thumper, it's a 4U 2-socket Opteron box with 48 SATA disks. You can have up to 48TB of disk on a single machine, although with RAID-5 volumes this would be somewhat smaller.

This blog posting discusses some of the ins and outs of using a Thumper as an iSCSI target.

Another option is the secondhand market. You can get re-certified SANs quite cheaply and many (even older ones such as Clariion CXs) have the option of slower SATA or FC-SATA disks. If you don't mind the power bills you can get second hand Symmetrix gear quite cheaply and that has no issues around licensing for the management software - you can operate it through the console on the controller.

Solution 3

Ahh, haven't got enough reputation points to leave comments yet, so it's a seperate post I'm afraid.

With regard to the HP Lefthand pricing, a good site to grab the pricing on the Lefthand kit without calling HP is the StorageMojo, the StorageMojo also has list prices for most other manufacturers.

You're looking at about £40k for a 2 node setup with a total of 24 x 450GB SAS disks (10TB raw capacity), including the basic hardware support for 3 years. Slap on another £6k if you want the Premium support. This gives you pretty good redundancy and all the software features like Thin Provisioning, SnapShots, Remote Copy, Multi site DR which normally cost extra. The nodes hardware are based on the HP ProLiant DL185 G5 server (so that's 12 disks per node).

It came in at half the cost of an equivalent NetApp solution.

I think with HP having bought them out, the LeftHand solution now offers pretty good bang for the buck with some proper Enterprise grade support behind it, important when things go pear shaped and you need assistance in the wee hours.

Solution 4

We set this up for less than $AU3000 (so, probably halve that for the US market):

• Dual Core 2.4Ghz
• 2x Adaptec SATA RAID cards
• 4x Gigabit NICs
• 4U Rackmount Case
• SATA Hotswap backplanes for case (total of 10x trays)
• Loaded her up with off-the-shelf Seagate SATAs (with 5 year warranties, but see this SF question if you plan on doing that)
• Installed FreeNAS and used as an iSCSI target (although you have your choice of connectivity)

Worked like a charm, cheap massive storage with connectivy out the wazoo.

Solution 5

Take a look at the Xiotech Emprise 5000. With their sata "diskpac"s you can get 16TB in 3U with redundant controllers and battery backed cache.

If you're willing to drop FC for iSCSI Dell's MD3000i or Equalogic products are the cheapest "name brand" SAN.

And if you really wanna get your freak on, check out Coraid's AoE. Though I'm not sure to what degree they meet your global presence requirements.

View more solutions

Comments

[ConcernedOfTunbridgeWells]

Could you elaborate on the requirements a bit - roughly how much disk do you need and what is the application. Would a NAS type architecture work or do you need block level storage with a clustered file system? For example, if it is a database, you can use some DBMS platforms with NFS shares (NFS provides adequate locking semantics for this but SMB/CIFS does not). In other cases (e.g. SQL Server) you would need block level storage. If it's just a shared pool of files, you may be better off with a NAS anyway.

[Toto]

What is cheap for you? What is "plentiful" storage for you? What is "highly available" for you? RAID? clustered NAS? ...?

[w00t]

:-O Too localized? Is the group wanting cheap but highly available storage that small?

[w00t]

@FlorenzKley, hardly cheap but interesting, thanks :)

[Florenz Kley]

@w00t - actually, dirt cheap compared to other SAN storage when expressed as $/GB. I admit the treshold is high with 27T as "entry level", but the price per unit of storage is ridiculous. Caveat: the 27TB model does not support iSCSI by itself (unless you are devious and hook up a couple of phased out boxes to attach over FC and offer iSCSI. Share and enjoy)

[Florenz Kley]

@mark-henderson: don't agree with it beeing "too localized". The shopping list in the question is exactly what's on lots of peoples minds right now. Care to elaborate how the dimensions of the question satisfy the narrowness criteria?

[MrGigu]

@FlorenzKley - it's a shopping question, which have been off topic for quite a long time. Please see the faq. Additionally, the question was asked almost 3 years ago, so any answers provided then will be totally out of date by now. Hence too localised (time-wise)

[reconbot]

That's EqualLogic. equallogic.com I recently went to a demo about these things, they look amazing, and easy to setup.

[Chopper3]

seconded! they're a great all-round box

[Dayton Brown]

+1 for "Get your Freak on" and AoE.

[w00t]

The Thumper is some very nice hardware, but it's still on the expensive side and it doesn't allow two controllers (well, that would be 2 mainboards). I'm still a fan though...

[w00t]

I must say, their XRaid is interesting, with it they seem to offer the same technology that made Drobo popular. This test on SmallNetBuilder reveals it to be best of class: smallnetbuilder.com/content/view/30725/75 Now the only problems are worldwide support and the non-rackmount formfactor, that could be a tough sell. Very interesting nonetheless.

[MrGigu]

+1 for the second hand market

[w00t]

well the problem we ran into in Australia was that one of our support guys had to have his credit card charged before they could ship a replacement drive, they would refund the money as soon as they received the disk back. Pretty tough for enterprise-level support :-/

[w00t]

This is the fun solution :-) Only problem is supportability, both from a software and hardware standpoint (OS patches, hardware compatibility, subtle driver failures, ...). It's also not highly available but given the low cost it is feasible to have multiple of these and to use host-based software RAID to make it highly available. This question has thought me that there is a cheap, consumer-hardware solution space and an expensive, enterprise solution space, with not much in between.

[pauska]

That's pretty harsh. Are there no resellers nearby who can provide disks?

[Matt Simmons]

The prices aren't on there, and I don't really feel like calling an HP salesman. How much do they run, roughly?

[Matt Simmons]

I looked into these. I disliked their crazy licensing terms. Extra costs for connecting over X number of machines, or if you had more than Y number of slices. Not cool.

[Zypher]

I havn't run into those. I've run into stupid things like if you hook it to vmware or more than 4 hosts they try and make you do the "Professional onsite install" but i just tell my rep i don't need it and he takes it off. Then again we have a giant enterprise account and I can get my rep to do pretty much what i want so YMMV.

[w00t]

Thanks for the explanation and the link! One question, when I look up SATA prices for Lefthand, I get P-2060-1TB07-HD: Spare NSM 2060 1.0TB SATA Drive, $2,380. Is this for 1 drive? Seriously? 1TB SATA drives normally go for 99$...

[Mannedood]

Fully loaded SATA dual server setup (about 4TB fully redundant) would run you about 35k or so.

[w00t]

Very interesting! They only have partners in the US though, so it's a no-go for us :( I wonder if VMWare will ever use this approach, every node you add to a VMWare cluster would add clustered storage.

[ConcernedOfTunbridgeWells]

NetApp seemed to do OK without redundant controllers for a while. I'm not sure if Solaris supports multipathing for its iScsi target.

[w00t]

good writeup in your link, interesting!

[rthomson]

+1 again for CORAID and AoE