Linux – Why does “tar -cSf file.tar source” run out of memory

backuplinuxtar

I'm trying to back up 66 gigabytes to a NAS by making a tarball, but "tar -cSf …" quits with a "memory exhausted" error after gigabyte 62. My Linux machine has a gigabyte of RAM and a gigabyte of swap space.

(edit) I tried it again about where tar gave up and tar quickly gave up again, so it looks like it may be having trouble dealing with a special file.

This data is surprisingly resistant to being backed up. rsync is 4 times slower than tar because the NAS isn't very fast and it quits in the middle with 'connection reset by peer'; 'cp' doesn't work well on the cifs share because it can't create the special files. Is there a better way?

Best Answer

-S is doing some checking for sparse files (those where not all file extents are actually physically allocated on disk). This could possibly be running out of memory. Try running it without the -S (compress it if you really want) and see if this fixes the problem.

tar cf foo.tar *

tar czf foo.tar.gz *

Related Solutions

Cron – When Does cron.daily Run

For the distributions you mention:

On CentOS 5.4 (Should be same for RHEL5)

grep run-parts /etc/crontab

01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly

So cron.daily runs at 04:02am.

Same on CentOS 4.8

Linux – What does Virtual memory size in top mean

Virtual memory isn't even necessarily memory. For example, if a process memory-maps a large file, the file is actually stored on disk, but it still takes up "address space" in the process.

Address space (ie. virtual memory in the process list) doesn't cost anything; it's not real. What's real is the RSS (RES) column, which is resident memory. That's how much of your actual memory a process is occupying.

But even that isn't the whole answer. If a process calls fork(), it splits into two parts, and both of them initially share all their RSS. So even if RSS was initially 1 GB, the result after forking would be two processes, each with an RSS of 1 GB, but you'd still only be using 1 GB of memory.

Confused yet? Here's what you really need to know: use the free command and check the results before and after starting your program (on the +/- buffers/cache line). That difference is how much new memory your newly-started program used.

Best Answer

Related Solutions

Cron – When Does cron.daily Run

Linux – What does Virtual memory size in top mean

Related Topic