How much resources should backup tools be using? ( issues with clBackup )

Question

My hosting provider makes backups of our servers using a tool called clBackup which pushes files over the network. Below is one ps output we took and reported to our provider:

top - 10:06:24 up 25 days,  3:47,  5 users,  load average: 6.63, 4.79, 4.23
Tasks: 357 total,   1 running, 355 sleeping,   0 stopped,   1 zombie
Cpu(s): 25.0%us,  0.6%sy,  0.0%ni, 72.5%id,  1.2%wa,  0.5%hi,  0.1%si,  0.0%st
Mem:  49447692k total, 49314632k used,   133060k free,    79628k buffers
Swap:  2097144k total,      288k used,  2096856k free, 40614172k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
20621 root      16   0  455m  65m  34m S 612.7  0.1 144:53.17 clBackup

We're experiencing performance issues on a production website hosted on these servers because of that tool. We know this for sure because whenever people complain our website is slow and we check the boxes, we can see the backup program running, whenever we kill that process the website is performing well again.

Apparently clBackup is somewhat good at eating up CPU (and memory) as we don't seem to be the only ones to have run into those issues (Link 1, Link 2, Link 3).

We're trying to get our hosting provider to understand that this performance problem isn't acceptable and point out that the high CPU usage isn't normal (more than 600%) but the advise we received was to exclude some large directories which don't really need to be backed up.

We're not very happy with this response (we don't want to spend resources in accommodating the backup tool for a service we pay for, we believe the backup tool should be designed to handle this situation) and are looking for help in the form of independent and objective answers from the ServerFault community, hoping this will help convince our provider to do more about it.

In order to make this question useful to everybody, we would like to raise few generic questions related to our problem that we hope experienced sysadmins will be able to answer.

Q1: From your own experience, what is an acceptable time to backup a 500GB file system of which 290GB is used?
Q2: From your own experience, what is an acceptable CPU usage for a backup utility of this kind?
Q3: Has anyone experienced similar issues with clBackup and did re-nicing the process or other mitigation measures help?

Solution 1

Yeah, that's pretty poor -- a backup program shouldn't be eating 6 cores to do it's job. I'd complain long, loud, and hard to the provider, and be prepared to leave for a provider who doesn't suck if they don't fix it.

To answer your questions:

1. From my own experience, it depends on too many factors to be able to give a single answer.
2. From my own experience, 10-20% of a core of a modern CPU should be more than enough to run a backup.
3. I've never touched clBackup, and I never will.

Solution 2

Yeah, running into the same issue ourselves. The only solution we have found is to tell our host to turn it off temporarily, and then we engage in a long issue thread that usually culminates with us being told to exclude directories, which really isn't a solution.

Sorry I don't have a solution, just wanted to let you know you're not alone.

Comments

[Zeyin]

We're struggling to come up with a solution for the performance issue of clBackup as well. Anyone knows what kind of stuff it does? We're thinking to get rid of it completely but don't know how important the process it. Cheers, Z.

[Max]

FYI never got to the bottom of the problem as changed jobs. clBackup seems to be a performance killer. If you can replace it with another backup solution that offers you a similar (or better) level of functionality AND improve performance, then do it. What you need to bear in mind is that if you get rid of clBackup, your provider might be less inclined to support you.