Wget an remote website index directory

file-transferwget

Is it possible to wget an entire index/parent directory with wget? Because I am trying to transfer images to a VPS of mine from my old shared host.

When I say an index/parent directory, I meant like this:

enter image description here

So basically I want wget to go through each file and download it until all have been downloaded.

Cheers.

Best Answer

you did mention there is ftp access, wget actually supports ftp out of the box:

wget -r -l0 --ftp-user=user --ftp-password=pass ftp://yourdomain.com/dir/*

Related Solutions

Linux – Copying a large directory tree locally? cp or rsync

I would use rsync as it means that if it is interrupted for any reason, then you can restart it easily with very little cost. And being rsync, it can even restart part way through a large file. As others mention, it can exclude files easily. The simplest way to preserve most things is to use the -a flag – ‘archive.’ So:

rsync -a source dest

Although UID/GID and symlinks are preserved by -a (see -lpgo), your question implies you might want a full copy of the filesystem information; and -a doesn't include hard-links, extended attributes, or ACLs (on Linux) or the above nor resource forks (on OS X.) Thus, for a robust copy of a filesystem, you'll need to include those flags:

rsync -aHAX source dest # Linux
rsync -aHE source dest  # OS X

The default cp will start again, though the -u flag will "copy only when the SOURCE file is newer than the destination file or when the destination file is missing". And the -a (archive) flag will be recursive, not recopy files if you have to restart and preserve permissions. So:

cp -au source dest

Wget to print errors, but nothing otherwise

There are very good answers in this question, be sure to check them out, but what I've done is this:

wget [wget options] 2>&1 | grep -i "failed\|error"

Best Answer

Related Solutions

Linux – Copying a large directory tree locally? cp or rsync

Wget to print errors, but nothing otherwise

Related Topic