Linux – The –reject feature in wget doesn’t seem to work

linuxmirroringwget

I am trying to mirror most of a ftp sever using wget.

This particular sever keeps old copies of the data files in a folder called 'backup' inside every subfolder. e.g. '/MaffiaOffShore/backup' '/VeryVeryDodgy/backup', which I don't want but my attempts at using –reject 'backup' and using –exclude-directories 'backup' and various other combinations, all have the same result, the whole sever is mirrored, including the backup folder.

Is this a bug or am I missing something?

Best Answer

The exclude directories option is:

--exclude-directories=List

not --exclude-directory as you have included in your question

from the wget man page

-X list
--exclude-directories=list
   Specify a comma-separated list of directories you wish to exclude
   from download (@pxref{Directory-Based Limits} for more details.)
   Elements of list may contain wildcards.

Related Solutions

Making `wget` not save the page

You can redirect the output of wget to /dev/null (or NUL on Windows):

wget http://www.example.com -O /dev/null

The file won't be written to disk, but it will be downloaded.

Wget doesn’t take variable http_proxy

Thanks for the debug logs.

So you're clearly not using up the env proxy settings you expect based on that output.

See http://www.gnu.org/software/wget/manual/wget.html#Proxies for some alternate methods of configuring the proxy settings.

It's not clear if you're in some kind of controlled environment, but it's also possible that the wget you're calling has been modified in some how -- perhaps there's a wrapper script that's stripping out the environment or disabling the proxy settings from being used (--no-proxy). Does file $(which wget) show it's an executable?

Best Answer

Related Solutions

Making `wget` not save the page

Wget doesn’t take variable http_proxy

Related Topic