Linux – Real CPU load with multiple cores

I would use rsync as it means that if it is interrupted for any reason, then you can restart it easily with very little cost. And being rsync, it can even restart part way through a large file. As others mention, it can exclude files easily. The simplest way to preserve most things is to use the -a flag – ‘archive.’ So:

rsync -a source dest

Although UID/GID and symlinks are preserved by -a (see -lpgo), your question implies you might want a full copy of the filesystem information; and -a doesn't include hard-links, extended attributes, or ACLs (on Linux) or the above nor resource forks (on OS X.) Thus, for a robust copy of a filesystem, you'll need to include those flags:

rsync -aHAX source dest # Linux
rsync -aHE source dest  # OS X

The default cp will start again, though the -u flag will "copy only when the SOURCE file is newer than the destination file or when the destination file is missing". And the -a (archive) flag will be recursive, not recopy files if you have to restart and preserve permissions. So:

cp -au source dest

The interesting part (in addition to MySQL being so busy...) is that majority of the CPU time seems to go to system, not user time.

Can you trigger this same kind of behaviour with some other program than MySQL? Try, for example, stress and make it consume the CPU with stress -c 128. For me that spawned 128 threads consuming 93% user, 7% system CPU (according to top). This in Fedora 15, anyway. If even stress makes your server consume mostly system time, something is very wrong.

Sometimes unexpected things like outdated server firmware can cause all kind of funky side-effects. Have you upgraded your server firmware? Also newish kernel features, like dynamic ticks can make things laggy under certain conditions. You may try disabling those by passing nohz=off as boot parameter to GRUB.