Linux – bash script returns “out of memory” in cron, but not in shell

bashcentoscronlinuxrsync

I'm running a nightly bash script to sync a remote folder (source) with a local folder (target). I've tested this script, based on rsync, and it works fine in a root shell. It takes time since there are hundred of gigs to copy but it works.

Once I use it in crontab my server runs out of memory.

My server has 8GB of RAM, 4GB of swap, and as I said, the script never goes OOM when manually ran from a shell. It's a default Centos 5.5 installation. I could split the load and sync all the 2nd level dirs in a find/for script, but I'd like to keep it simple and only sync the top level directories.

I cannot make too many tests since this server is used to host websites and other services and I cannot afford to hang it just for testing purpose. Do you know a setting that could allow cron to finish this job normally ?

#!/bin/bash

BACKUP_PATH="/root/scripts/backup"

rsync -av --delete /net/hostname/source/ /export/target/ >     $BACKUP_PATH/backup_results_ok 2>  $BACKUP_PATH/backup_results_error

edit: cron configuration is default, as /etc/security/limits.conf, which is all commented out.

Best Answer

Even though limits.conf is commented out, I suggest that you test just to make sure. One way would be to create a cronjob that just contains something like "ulimit -a | Mail -s 'limits' me@example.com" to have the info emailed to you. Once you know what the limits are, then you can just reset them in the shell script that actually runs the rsync:

#!/bin/bash
ulimit -d unlimited
ulimit -m unlimited
ulimit -s unlimited
rsync [...]

Related Solutions

Linux – How to run a command multiple times, using bash shell

I don't think a command or shell builtin for this exists, as it's a trivial subset of what the Bourne shell for loop is designed for and implementing a command like this yourself is therefore quite simple.

Per JimB's suggestion, use the Bash builtin for generating sequences:

for i in {1..10}; do command; done

For very old versions of bash, you can use the seq command:

for i in `seq 10`; do command; done

This iterates ten times executing command each time - it can be a pipe or a series of commands separated by ; or &&. You can use the $i variable to know which iteration you're in.

If you consider this one-liner a script and so for some unspecified (but perhaps valid) reason undesireable you can implement it as a command, perhaps something like this on your .bashrc (untested):

#function run
run() {
    number=$1
    shift
    for i in `seq $number`; do
      $@
    done
}

Usage:

run 10 command

Example:

run 5 echo 'Hello World!'

Bash – Standard place for user defined bash_completion.d scripts

Use a ~/.bash_completion file.

From the Bash Completion FAQ:

Q. How can I insert my own local completions without having to
reinsert them every time you issue a new release?

A. Put them in ~/.bash_completion, which is parsed at the end of the
main completion script. See also the next question.

Q. I author/maintain package X and would like to maintain my own
completion code for this package. Where should I put it to be sure
that interactive bash shells will find it and source it?

A. Install it in one of the directories pointed to by
bash-completion's pkgconfig file variables. There are two
alternatives: the recommended one is 'completionsdir' (get it with
"pkg-config --variable=completionsdir bash-completion") from which
completions are loaded on demand based on invoked commands' names,
so be sure to name your completion file accordingly, and to include
for example symbolic links in case the file provides completions
for more than one command. The other one which is present for
backwards compatibility reasons is 'compatdir' (get it with
"pkg-config --variable=compatdir bash-completion") from which files
are loaded when bash_completion is loaded.

Best Answer

Related Solutions

Linux – How to run a command multiple times, using bash shell

Bash – Standard place for user defined bash_completion.d scripts

Related Topic