Determine if a File is Being Written to in Linux

bashlinuxtar

I need to deploy an automated process (via 1 min cron script) that looks for tar files in a specific directory. If a tar file is found, it is untarred to the appropriate location and then the tar file is deleted.

The tar files are automatically copied to this server over SSH from another server. In some cases, the tar files are extremely large, with lots of files.

The problem that I am expecting to run into: If it takes > 1 minute for the tar file to be copied to the server, and the cron script runs once every minute, it's going to see the .tar.gz file and try to do untar it, even though the tar file is still in the process of being written to.

Is there any way (via bash commands) to test if a file is currently being written to, or if it's only a partial file, etc?

One alternative I was thinking of was to have the file be copied as a different file extension (like .tar.gz.part) and then renamed to .tar.gz after the transfer is complete. But I figured I'd try to figure out if there is simply a way to determine if the file is whole at the command line first… Any clues?

Best Answer

You are on the right track, renaming the file is an atomic operation, so performing the rename after upload is simple, elegant and not error prone. Another approach I can think of is to use lsof | grep filename.tar.gz to check if the file is being accessed by another process.

Related Solutions

Linux – Help on writing Linux script to backup and ftp files to another server

I would not recommend using FTP for any files you care about (particularly database backups if that database contains customer information), but that is ultimately up to you.

To start with, script the creation of your backups on the Linux server with something like the following:

#!/bin/bash
# 
# Call script with:
# /cron-mysql_bkup.sh mysql_hostname mysql_username mysql_password /backup/directory
# 
if [ -n "$1" ]; then
 MYSQL_HOST="$1"
else
 echo "MYSQL_HOST not specified."
 exit 1
fi
if [ -n "$2" ]; then
 MYSQL_USER="$2"
else
 echo "MYSQL_USER not specified."
 exit 1
fi
if [ -n "$3" ]; then
 MYSQL_PASS="$3"
else
 echo "MYSQL_PASS not specified."
 exit 1
fi
if [ -n "$4" ]; then
 DIR_BKUP="$4"
 mkdir -p ${DIR_BKUP}
else
 echo "DIR_BKUP not specified."
 exit 1
fi
# 
# Generate filename
# 
BACKUP_FILENAME=`date "+%Y-%m-%d"`".sql"
# 
# Dump all databases
# 
mysqldump -h${MYSQL_HOST} -u${MYSQL_USER} -p${MYSQL_PASS} --all-databases > ${DIR_BKUP}/${MYSQL_HOST}/${BACKUP_FILENAME}
# 
# Write status of the operation to the mysql.log file in the backup directory
# 
if [ $? -ne 0 ]; then
 echo `date`" - ERROR creating backups of host "${MYSQL_HOST} >> ${DIR_BKUP}/mysql.log
else
 echo `date`" - MySQL Backup complete for "${MYSQL_HOST} >> ${DIR_BKUP}/mysql.log
 BACKUP_TAR_NAME=${BACKUP_FILENAME}".tar.gz"
 tar czf ${DIR_BKUP}/${MYSQL_HOST}/${BACKUP_TAR_NAME} ${DIR_BKUP}/${MYSQL_HOST}/${BACKUP_FILENAME} && rm ${DIR_BKUP}/${MYSQL_HOST}/${BACKUP_FILENAME}
fi

This script will:

Check to see whether all the required parameters were set
Dump all MySQL databases to a file named YYYY-mm-dd.sql under the backup directory specified (this directory will be created if it does not exist)
GZIP the file and, if the GZIP operation succeeds, delete the original SQL file

This script should be fairly straightforward for you to customize if you wish to - see the Advanced Bash Scripting Guide to learn more about doing so.

... once your databases are backed up and can be found in a specific location, why not automate backup retrieval with WinSCP instead of transferring your files over FTP?

Linux how to move a large file

If you have a network connection then just do a ssh pipe.

ssh user@host "cat > file.remote" < file.local

If you need to split the file for media then use split to break it up and cat to fuse it back together.

split -b 1G file.local filexfer
cat filexfer* > file.remote

Note that all this assumes that the remote filesystem is actually capable of storing files that large.

Related Topic