Keep rsync from removing unfinished source files

rsyncstorageweb-crawler

I have two machines, speed and mass. speed has a fast Internet connection and is running a crawler which downloads a lot of files to disk. mass has a lot of disk space. I want to move the files from speed to mass after they're done downloading. Ideally, I'd just run:

$ rsync --remove-source-files speed:/var/crawldir .

but I worry that rsync will unlink a source file that hasn't finished downloading yet. (I looked at the source code and I didn't see anything protecting against this.) Any suggestions?

Best Answer

It seems to me the problem is transferring a file before it's complete, not that you're deleting it.

If this is Linux, it's possible for a file to be open by process A and process B can unlink the file. There's no error, but of course A is wasting its time. Therefore, the fact that rsync deletes the source file is not a problem.

The problem is rsync deletes the source file only after it's copied, and if it's still being written to disk you'll have a partial file.

How about this: Mount mass as a remote file system (NFS would work) in speed. Then just web-crawl the files directly.

Related Solutions

Windows – Use WMI to remove a page file

Found it.

There is a .Delete() method that does the trick.

PS D:\> $pf[1].Delete()
PS D:\> gwmi win32_pagefilesetting

                            MaximumSize Name                                    Caption
                            ----------- ----                                    -------
                                   4096 c:\pagefile.sys                         c:\ 'pagefile.sys'

Done.

Copying files using rsync from remote server to local machine

From your local machine:

rsync -chavzP --stats user@remote.host:/path/to/copy /path/to/local/storage

From your local machine with a non standard ssh port:

rsync -chavzP -e "ssh -p $portNumber" user@remote.host:/path/to/copy /local/path

Or from the remote host, assuming you really want to work this way and your local machine is listening on SSH:

rsync -chavzP --stats /path/to/copy user@host.remoted.from:/path/to/local/storage

See man rsync for an explanation of my usual switches.

Best Answer

Related Solutions

Windows – Use WMI to remove a page file

Copying files using rsync from remote server to local machine

Related Topic