Windows – How to download all the files in a directory with cURL

bashcurlshellwindows

I am using cURL to try to download all files in a certain directory.

Here's what my list of files looks like:

List of Files

I have tried to do in bash script: iiumlabs.[].csv.pgp and iiumlabs* and I guess cURL is not big on wildcards.

curl -u login:pass ftp.myftpsite.com/iiumlabs* -O

Question: How do I download this directory of files using cURL?

Best Answer

Curl does not support recursive download. Use wget --mirror --no-parent [URL]

EDIT: For SSH, from the man page of curl:

Get a file from an SSH server using SFTP:

    curl -u username sftp://shell.example.com/etc/issue

Get a file from an SSH server using SCP using a private key to authenticate:

    curl -u username: --key ~/.ssh/id_dsa --pubkey ~/.ssh/id_dsa.pub \
            scp://shell.example.com/~/personal.txt

Related Solutions

Curl – download file range

I can't think of a way to do this and re-use the connection, but you could do it with a for loop and a new connection each time. In bash, for instance:

for i in {1..100}; do curl -R -O -z /dir/file${i}.png http://somesite.com/file${i}.png; done

Linux – curl –ftp-ssl to grab all files in remote directory

AFAIK, there is no such option to download a directory with curl, so you must get the listing first and pipe it to curl to download file by file, something like this:

$ curl -s ftp://user:pass@IP/path/to/folder/ | \
    grep -e '^-' | awk '{ print $9 }' | \
        while read f; do \
            curl -O ftp://user:pass@IP/path/to/folder/$f; \
        done

Best Answer

Related Solutions

Curl – download file range

Linux – curl –ftp-ssl to grab all files in remote directory

Related Topic