How much resources should backup tools be using? ( issues with clBackup )

backupcpu-usagefilesystemsmemory usageperformance

My hosting provider makes backups of our servers using a tool called clBackup which pushes files over the network. Below is one ps output we took and reported to our provider:

top - 10:06:24 up 25 days,  3:47,  5 users,  load average: 6.63, 4.79, 4.23
Tasks: 357 total,   1 running, 355 sleeping,   0 stopped,   1 zombie
Cpu(s): 25.0%us,  0.6%sy,  0.0%ni, 72.5%id,  1.2%wa,  0.5%hi,  0.1%si,  0.0%st
Mem:  49447692k total, 49314632k used,   133060k free,    79628k buffers
Swap:  2097144k total,      288k used,  2096856k free, 40614172k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
20621 root      16   0  455m  65m  34m S 612.7  0.1 144:53.17 clBackup

We're experiencing performance issues on a production website hosted on these servers because of that tool. We know this for sure because whenever people complain our website is slow and we check the boxes, we can see the backup program running, whenever we kill that process the website is performing well again.

Apparently clBackup is somewhat good at eating up CPU (and memory) as we don't seem to be the only ones to have run into those issues (Link 1, Link 2, Link 3).

We're trying to get our hosting provider to understand that this performance problem isn't acceptable and point out that the high CPU usage isn't normal (more than 600%) but the advise we received was to exclude some large directories which don't really need to be backed up.

We're not very happy with this response (we don't want to spend resources in accommodating the backup tool for a service we pay for, we believe the backup tool should be designed to handle this situation) and are looking for help in the form of independent and objective answers from the ServerFault community, hoping this will help convince our provider to do more about it.

In order to make this question useful to everybody, we would like to raise few generic questions related to our problem that we hope experienced sysadmins will be able to answer.

Q1: From your own experience, what is an acceptable time to backup a 500GB file system of which 290GB is used?
Q2: From your own experience, what is an acceptable CPU usage for a backup utility of this kind?
Q3: Has anyone experienced similar issues with clBackup and did re-nicing the process or other mitigation measures help?

Best Answer

Yeah, that's pretty poor -- a backup program shouldn't be eating 6 cores to do it's job. I'd complain long, loud, and hard to the provider, and be prepared to leave for a provider who doesn't suck if they don't fix it.

To answer your questions:

  1. From my own experience, it depends on too many factors to be able to give a single answer.
  2. From my own experience, 10-20% of a core of a modern CPU should be more than enough to run a backup.
  3. I've never touched clBackup, and I never will.
Related Topic