Linux – Using Rsync include and exclude options to include directory and file by pattern

darwinlinuxrsyncunix

I'm having problems getting my rsync syntax right and I'm wondering if my scenario can actually be handled with rsync. First, I've confirmed that rsync is working just fine between my local host and my remote host. Doing a straight sync on a directory is successful.

Here's what my filesystem looks like:

uploads/
  1260000000/
    file_11_00.jpg
    file_11_01.jpg
    file_12_00.jpg
  1270000000/
    file_11_00.jpg
    file_11_01.jpg
    file_12_00.jpg
  1280000000/
    file_11_00.jpg
    file_11_01.jpg
    file_12_00.jpg

What I want to do is run rsync only on files that begin with "file_11_" in the subdirectories and I want to be able to run just one rsync job to sync all of these files in the subdirectories.

Here's the command that I'm trying:

rsync -nrv --include="**/file_11*.jpg" --exclude="*" /Storage/uploads/ /website/uploads/

This results in 0 files being marked for transfer in my dry run. I've tried various other combinations of --include and --exclude statements, but either continued to get no results or got everything as if no include or exclude options were set.

Anyone have any idea how to do this?

Best Answer

The problem is that --exclude="*" says to exclude (for example) the 1260000000/ directory, so rsync never examines the contents of that directory, so never notices that the directory contains files that would have been matched by your --include.

I think the closest thing to what you want is this:

rsync -nrv --include="*/" --include="file_11*.jpg" --exclude="*" /Storage/uploads/ /website/uploads/

(which will include all directories, and all files matching file_11*.jpg, but no other files), or maybe this:

rsync -nrv --include="/[0-9][0-9][0-9]0000000/" --include="file_11*.jpg" --exclude="*" /Storage/uploads/ /website/uploads/

(same concept, but much pickier about the directories it will include).

Related Solutions

Unix – Use grep –exclude/–include syntax to not grep through certain files

Use the shell globbing syntax:

grep pattern -r --include=\*.cpp --include=\*.h rootdir

The syntax for --exclude is identical.

Note that the star is escaped with a backslash to prevent it from being expanded by the shell (quoting it, such as --include="*.cpp", would work just as well). Otherwise, if you had any files in the current working directory that matched the pattern, the command line would expand to something like grep pattern -r --include=foo.cpp --include=bar.cpp rootdir, which would only search files named foo.cpp and bar.cpp, which is quite likely not what you wanted.

Update 2021-03-04

I've edited the original answer to remove the use of brace expansion, which is a feature provided by several shells such as Bash and zsh to simplify patterns like this; but note that brace expansion is not POSIX shell-compliant.

The original example was:

grep pattern -r --include=\*.{cpp,h} rootdir

to search through all .cpp and .h files rooted in the directory rootdir.

Linux – How to redirect output to a file and stdout

The command you want is named tee:

foo | tee output.file

For example, if you only care about stdout:

ls -a | tee output.file

If you want to include stderr, do:

program [arguments...] 2>&1 | tee outfile

2>&1 redirects channel 2 (stderr/standard error) into channel 1 (stdout/standard output), such that both is written as stdout. It is also directed to the given output file as of the tee command.

Furthermore, if you want to append to the log file, use tee -a as:

program [arguments...] 2>&1 | tee -a outfile

Best Answer

Related Solutions

Unix – Use grep –exclude/–include syntax to not grep through certain files

Linux – How to redirect output to a file and stdout

Related Topic