I need to download 100 GB of files (it’s in about 150 files) within a 7 day period before they expire. The download is rate-limited by the host so it takes MUCH longer than the theoretical transfer rate based on normal Internet speeds. I have a script of curl http://curl.haxx.se/docs/manpage.html commands – one line per file.
I had the idea of automatically spinning up n EC2 instances, executing the command and FTPing the files to a central location, then shutting down the machines. How would I do this?
I don't care whether it's Linux or Windows.
Best Answer
I would recommend using one of Canonical's AMIs in conjuction with cloud-init
https://help.ubuntu.com/community/CloudInit
Look for the section about "User-Data Scripts" where you can use the --user-data-file to auto-launch a script on startup. Thus, you can just put all the commands you want to run in the script and put "shutdown -h now" at the end of the script.