ZFS and FreeNAS Performance

Question

I have just set up an HP Micro Server N40L as a FreeNAS with 4 2tb drives in a RAIDZ. I am getting around 30-40MB/sec, with occasional bursts of 50MB/sec., reads and writes that are similar to the low end of that range. As far as I can tell the CPU is not overworked and when I did a file copy internally from one folder in the RAIDZ to another, I hit over 80MB/sec.

I am wondering what might be the best approach to improving read and write speeds. Tweaking my GigE network, adding a ZFS cache drive, etc.?

Solution

I have the same Server and same disks - the issue is using 4 disks in a RAIDZ1 Pool, using 3 or 5 disks is fine.

The recommendations for RAID-Z disk configurations:

• Start a single-parity RAIDZ (raidz) configuration at 3 disks (2+1)
• Start a double-parity RAIDZ (raidz2) configuration at 6 disks (4+2)
• Start a triple-parity RAIDZ (raidz3) configuration at 9 disks (6+3) N+P) with P = 1 (raidz), 2 (raidz2), or 3 (raidz3) and N equals 2, 4, or 6
• The recommended number of disks per group is between 3 and 9.
• If you have more disks, use multiple groups.

from: http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#RAIDZ_Configuration_Requirements_and_Recommendations

This forum post http://forums.freenas.org/archive/index.php/t-5201.html originally pointed me in the right direction - I saw similar performance drop off when I built a 4 disk pool, keeping to 3 disks and it flew again.

I tend to get ~ 80MB/s over SMB/CIFS (if I'm lucky), but for shifting larger files I tend to use FTP and peak over 110MB/s.

Comments

[ravi yarlagadda]

You'll need to take a pretty exhaustive approach to finding the bottleneck before you can figure out what to change to increase speed. What protocol are you using for the remote file sharing? Is the traffic going through equipment that's capable of carry a full gig the whole way? How about the client system's performance? And is there potentially any extra load or any disk thrashing occurring at the same time as the slower transfers?

[Timothy R. Butler]

I did some more testing with my MacBook Pro (Mid-2009) connected directly to the D-Link DGS2208 Gig-E switch that the Micro Server is plugged into. With the volume mounted via AFP, I timed cp in bash and found that uploading improved to 56.1MBps. Downloads came out at 71.07MBps. (In each case, my MBP is copying to or from its Crucial M4 SSD.) Uploads were unsurprisingly slower when I tried scp. Server load averages seemed to peak at 0.45, 0.15, 0.10. Cp'ing on the MBP drive to itself, I hit 205.15MBps. On the server copying the file to itself hit 273.64MBps (!), so my method is clearly flawed.

[ravi yarlagadda]

Well, the theoretical peak of a GigE link is 125MB/s, which trims down for AFP protocol overhead, TCP overhead and ACKing, IP overhead, and ethernet overhead. Getting around 80 MB/s is pretty respectable for a GigE link.

[Alexy Durand]

How much memory does your N40L have? The basic N40L comes with 2GB memory, while freenas recommends a system with at least 6GB of memory. doc.freenas.org/index.php/Hardware_Requirements