Performance problem with QNAP TS-410 NAS

Question

We have a QNAP TS-410 in our office. Its hardware is:

• 256MB DDR2 RAM
• CPU Marvell 6281 800MHz
• 4 x 1.4GB Seagate SATA disks mounted on RAID5

We thought it would be sufficient to make backups for our servers, but we have encountered several problems:

1. It takes about 5 hours to make a daily rsync between two 100GB directories. The directory tree is big and complex, because it's our production server with 200 websites. But changes should be minor. And network bandwidth is certainly not a problem (we have made tests with regular files upload/download).
2. Once the rsync is made and all changes has been transferred, it takes more than 15 hours to make a rotation of these backups. We are using a cp -al approach to create hard links instead of copying.
3. The built-in program vs_refresh takes almost a whole day to achieve its task.

We think every problem is directly related with the complexity and size of the directory tree. Imagine two hundred websites, being each one of them a Joomla, or a WordPress, or a Moodle, etc.

We think it is relatively normal for rsync and cp to take a certain amount of time. But so much time? Is this an expected and normal behaviour?

If so, what can be done to improve the NAS performance? Maybe increasing the RAM to avoid swap replacements?

Thanks!

Solution

There have been IO performance issues reported in the QNAP forum where IO throughput is reduced under certain conditions (free disk space and free RAM). The related issue is dedicated to linux 2.6.33.2 kernel. This kernel is used in all QNAP NAS. In case the above mentioned io perf. issue falls in this category, then a workaround (that that is reported to work) is to get more free disk space (80%+) as well as 180MB of free avail RAM.

the detailed thread is http://forum.qnap.com/viewtopic.php?f=25&t=51741

Would be good to know if your issue is same as root cause of the reported kernel issue is not fully understood - thus there are many other threads in the qnap forum that report similar behaviour. but since today there is no good understanding nor a patch available. newer kernels do not have this issue any more.

Comments

[Steven]

Is the network gigabit? Can you run top on the NAS? Is the box swapping? Is the cpu at 100% the entire time? I wouldn't be surprised if you're CPU and RAM bound on the NAS. For the rotation, how many copies are there?

[Rusell Jane Quitain]

The network is not Gigabit, because we make remote backups (from the production server to a remote one) over a 10 mbps ADSL connection Yes, I can run top and it's swapping but not reaching 100% CPU. We mantain a 60 days rotation schema.

[Steven]

In top, what is the io wait (wa) at while doing the backup? I'm not surprised a 60 day rotation is taking 15 hours. Then you can run iostat -x 5.

[Rusell Jane Quitain]

Curious thing is that the rotation process is pretty fast (it is just a directory rename). The problem is recycling the last discarded backup (day 60, moved to a tmp directory) to make the cp -al of new backup faster. I'll check the iostat and tell something about it.

[Rusell Jane Quitain]

dstat revealed that while a backup rotation, IO values are between 220 and 350. I don't know what these values mean exactly. I'm asking a new question regarding this.

[Rusell Jane Quitain]

IO values are rising to a range between 400 and 900. I'm starting to get worried

[Rusell Jane Quitain]

Indeed, my kernel version is exactly 2.6.33.2 However, QNAP claims to be a bug affecting to firmware up to 3.5.2 but mine is 3.6. The filesystem has a lot of directories and files, so I think the problem is somehow related as well.