Linux – wget recursive download, but I don’t want to follow all links

linuxmirrormirror-sitewget

I'm trying to mirror a website using wget, but I don't want to download lots of files, so I'm using wget's --reject option to not save all the files. However wget will still download all the files and then remove the file afterwards if it matches my reject option.

Is there some way to tell wget not to follow certain links if they match some shell wildcard? If wget can't do this, is there some other common linux command that can do this?

Best Answer

You might also try HTTrack which has, IMO, more flexible and intuitive include/exclude logic. Something like this...

httrack "https://example.com" -O ExampleMirrorDirectory \
"-*" \
"+https://example.com/images/*" \
"-*.swf"

The rules will be applied in order, and will override previous rules...

Exclude everything
But include https://example.com/images/*
But exclude anything ending in swf

Related Solutions

Linux – wget: how to download a file, whose url params changes dynamically, once only

Write a local proxy server that modifies the responses sent to wget.

Assuming your URLs are in links such as:

<a href="/path/to/mainpage.do?cx=0.0123412341234">

Then you can run a Ruby proxy server like this:

require 'webrick/httpproxy'
s = WEBrick::HTTPProxyServer.new(
   :Port => 2200,
   :ProxyContentHandler => Proc.new{|req,res|
      res.body.gsub!(/mainpage.do?cz=[0-9\.]*/, "mainpage.do")
   } 
)  
trap("INT"){ s.shutdown }
s.start

Ssh – “wget –mirror” Alternative

rsync is the standard utility for this:

rsync -avz -e ssh source/ user@destServer:/dest/

a For archive, keeps permissions, type stamps, etc.
v for verbose
e ssh , use over ssh
z for compression, if you want that. It won't try to re-compress archive (zip) files.

rsync generally comes with Linux distributions. It also meets your requirement of not transferring things that have not changed.

Best Answer

Related Solutions

Linux – wget: how to download a file, whose url params changes dynamically, once only

Ssh – “wget –mirror” Alternative

Related Topic