Amazon S3 – How to Copy Large Data Between Buckets in Different Regions

(Improving the solution of Shishir)

• Save the following script in a file (I named the file s3Copy.sh)

path=$1 # the path of the directory where the files and directories that need to be copied are located
s3Dir=$2 # the s3 bucket path

for entry in "$path"/*; do
    name=`echo $entry | sed 's/.*\///'`  # getting the name of the file or directory
    if [[ -d  $entry ]]; then  # if it is a directory
        aws s3 cp  --recursive "$name" "$s3Dir/$name/"
    else  # if it is a file
        aws s3 cp "$name" "$s3Dir/"
    fi
done

• Run it as follows:
  /PATH/TO/s3Copy.sh /PATH/TO/ROOT/DIR/OF/SOURCE/FILESandDIRS PATH/OF/S3/BUCKET
  For example if s3Copy.sh is stored in the home directory and I want to copy all the files and directories located in the current directory, then I run this:
  ~/s3Copy.sh . s3://XXX/myBucket

You can easily modify the script to allow for other arguments of s3 cp such as --include, --exclude, ...

I find AWS pricing really complicated. As you say "Transfers between S3 buckets or from S3 to any service(s) within the same region are free." as stated here on the AWS Pricing page: https://aws.amazon.com/s3/pricing/

However on the Simple Monthly calculator on the S3 section it does suggest there is a small cost for copy actions as well as Get and Put http://calculator.s3.amazonaws.com/index.html

Try using the calculator to get an estimated cost for you copy operations.