Ubuntu – How to free memory after rsync

backupmemoryrsyncUbuntu

I need to backup data from a web server with 10GB of RAM (Ubuntu).
To backup data I use Rsync.

My first test was to use one rsync for all website, but it uses 5GB.
So I decided to split rsync but it still uses 5GB.
In both of theses cases, it consume to much memory and are not always completely free after the process

If I manualy drop the cache between rsync, It only uses between 150Mb and 500Mb completely free the memory after the process. (And free 50%-70% memory of the server)

for CURDIR in "$ROOTDIR"*
do
    echo "Start $CURDIR"    
    rsync -aHh --stats --compress --delete "$CURDIR" --link-dest="saveofyesterday" "saveoftoday"    
    echo "Clear"
    sync
    echo 3 > /proc/sys/vm/drop_caches
    echo "Finish $CURDIR."
done

It's seems to be not recommend.
What is the proper way to achieve this ?

Best Answer

I'd suggest looking into fadvise patch for rsync. Purpose is to preserve disk cache while backing up. Effect you should see this memory you consider as 'used' will not be 'used'. In reality, that memory you consider used, it is used but it is used for disk cache and will be released if needed by programs. If system is low on memory eventually all disk cache will be evicted from memory in order to try and provide memory to needy apps. This is bad, especially on busy systems which don't have extremely fast disk sub system because once disk cache is evicted from memory all that once cached data will need to be read from disk again which will cause huge IO load and wait times.

Dropping cache manually is just as bad and will get your server in tough spot. This is not advisable route to go. Memory management is not something your script should worry about, kernel takes care of that part.

Related Topic