Linux – Using rsync to rename files during copying with –files-from

It seems to me the problem is transferring a file before it's complete, not that you're deleting it.

If this is Linux, it's possible for a file to be open by process A and process B can unlink the file. There's no error, but of course A is wasting its time. Therefore, the fact that rsync deletes the source file is not a problem.

The problem is rsync deletes the source file only after it's copied, and if it's still being written to disk you'll have a partial file.

How about this: Mount mass as a remote file system (NFS would work) in speed. Then just web-crawl the files directly.

Use:

find . 2>/dev/null > files_and_folders

This hides not just the Permission denied errors, of course, but all error messages.

If you really want to keep other possible errors, such as too many hops on a symlink, but not the permission denied ones, then you'd probably have to take a flying guess that you don't have many files called 'permission denied' and try:

find . 2>&1 | grep -v 'Permission denied' > files_and_folders

If you strictly want to filter just standard error, you can use the more elaborate construction:

find . 2>&1 > files_and_folders | grep -v 'Permission denied' >&2

The I/O redirection on the find command is: 2>&1 > files_and_folders |. The pipe redirects standard output to the grep command and is applied first. The 2>&1 sends standard error to the same place as standard output (the pipe). The > files_and_folders sends standard output (but not standard error) to a file. The net result is that messages written to standard error are sent down the pipe and the regular output of find is written to the file. The grep filters the standard output (you can decide how selective you want it to be, and may have to change the spelling depending on locale and O/S) and the final >&2 means that the surviving error messages (written to standard output) go to standard error once more. The final redirection could be regarded as optional at the terminal, but would be a very good idea to use it in a script so that error messages appear on standard error.

There are endless variations on this theme, depending on what you want to do. This will work on any variant of Unix with any Bourne shell derivative (Bash, Korn, …) and any POSIX-compliant version of find.

If you wish to adapt to the specific version of find you have on your system, there may be alternative options available. GNU find in particular has a myriad options not available in other versions — see the currently accepted answer for one such set of options.