SAS / SATA max number of connected disks vs performance

Question

I would like to understand following things:

When I have a motherboard with SATA 6Gb/s and 8 ports, does it mean that every single port is able to provide 6Gb/s speed to disk that is connected over SATA cable, or is 6Gb/s distributed evenly across all connected disks?

Is it possible to connect more than 8 disks if motherboard has only 8 slots? (Like multiple disks on 1 SATA cable?)

Regarding SAS disks, if I buy a disk controller with 4 ports that say it supports up to 64 disks, how do I connect these extra disks when there are only 4 ports?

Solution 1

SATA/SAS are not like hubs, rather they work much like a network switch. This means that a single SATA/SAS port provides the entire advertised bandwidth, independently from the other ports.

In other words, a chipset providing 4x SATA 3.0 ports (6 Gb/s each) has a "SATA backplane" switching capability of 4x6 Gb/s = 24 Gb/s. For a 4x SAS 3.0 ports (12 Gb/s each), the "SAS backplane" switching capability is 4x12 Gb/s = 48 Gb/s

Does this means that all the available raw bandwidth really is usable? Well, no. Problem is that these SATA/SAS ports must be connected by some other upstream bus, which will likely affect (reduce) the real available bandwidth.

Take, for example, the previous-gen Z97 Intel chipset: it has 6x SATA 3.0 ports, for a total of 36 Gb/s or 4.5 GB/s. This is a very respectable I/O transfer rate for a consumer product. However, the disk controller (integrated into the chipset) connects to the CPU using a much narrow (albeit full-duplex) 2 GB/s link. In other words, you will never reach the theoretical 4.5 GB/s peak of the SATA links.

Another complication is represented by SATA port multiplier or SAS port expander. What are these things? Well, they are what they claim to be: port multipliers/expanders to connect more drive to a single SATA/SAS port. You can think them as "mini-controller" which use a single SATA/SAS port as upstream bus and provide multiple ports to downstream devices.

This means that disks connected after a SATA port multiplier / SAS expander are limited by the bandwidth provided by the single upstream port - 1x SATA 6 Gb/s or SAS 3.0 12 Gb/s port. You can combine multiple upstream port to provide added performance, but the basic problem remains: you share multiple downstream port connected by few upstream links.

Let use your 4x SAS 3.0 controller as an example: using SAS expander you can link multiple drives to every single upstream SAS port - say 16x downstream SAS 3.0 links for each 1x SAS upstream link, for a grand total of 64x available SAS 3.0 ports / devices. Even if you populate all 64 SAS ports with lightning fast SSD, your total transfer rate will be limited by the 4x SAS links used to connect the downstream expander. So, from a theoretical 64x12 Gb/s (768 Gb/s or 96 GB/s), your real peak transfer rate will be 4x12 Gb/s (48 Gb/s or 6 GB/s).

Things are not so bad, though. After all, it is very difficult to hit all the disks at the same time. This is the reason behind multiplier/expander existance: to privilege disk capacity (and easy trace routing) vs raw performance.

Solution 2

It is 6 Gb/s per port, with 8 ports you can get up to 48 Gb/s total.

For SAS controllers and disks, you can use "SAS Expanders" to connect one SAS port to multiple drives. In this case, all drives connected to one SAS port share the total 6 Gb/s bandwidth.

For this reason, for maximum performance we avoid SAS expanders and use controllers like the LSI MegaRAID SAS 9280-24i4e with 24 independent SAS ports. (But it is a $1200+ controller, $1400+ once you get the battery on it).

Comments

[shodanshok]

Sure. You should combine that card with something similar to this

[Klaws]

Note that SATA port multiplieres cannot be cascaded. And that not all SATA controller support port multipliers...and even if they do, the driver may not. Or worse, hardware and drivers support it, but are buggy. SiI3132-absed controllers work quite well with port multipliers (under Windows and FreeBSD, as far as I can tell), but they max out at 150MByte/s (less if only one channel is used, I vaguely remember something like 120MByte/s).