The AWS CLI now supports the --query parameter which takes a JMESPath expressions.
This means you can sum the size values given by list-objects using sum(Contents[].Size) and count like length(Contents[]).
This can be be run using the official AWS CLI as below and was introduced in Feb 2014
aws s3api list-objects --bucket BUCKETNAME --output json --query "[sum(Contents[].Size), length(Contents[])]"
I was struggling with this, too, but I found an answer over here https://stackoverflow.com/a/17162973/1750869 that helped resolve this issue for me. Reposting answer below.
You don't have to open permissions to everyone. Use the below Bucket policies on source and destination for copying from a bucket in one account to another using an IAM user
Bucket to Copy from – SourceBucket
Bucket to Copy to – DestinationBucket
Source AWS Account ID - XXXX–XXXX-XXXX
Source IAM User - src–iam-user
The below policy means – the IAM user - XXXX–XXXX-XXXX:src–iam-user has s3:ListBucket and s3:GetObject privileges on SourceBucket/* and s3:ListBucket and s3:PutObject privileges on DestinationBucket/*
On the SourceBucket the policy should be like:
{
"Id": "Policy1357935677554",
"Statement": [
{
"Sid": "Stmt1357935647218",
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::SourceBucket",
"Principal": {"AWS": "arn:aws:iam::XXXXXXXXXXXX:user/src–iam-user"}
},
{
"Sid": "Stmt1357935676138",
"Action": ["s3:GetObject"],
"Effect": "Allow",
"Resource": "arn:aws:s3::: SourceBucket/*",
"Principal": {"AWS": "arn:aws:iam::XXXXXXXXXXXX:user/src–iam-user"}
}
]
}
On the DestinationBucket the policy should be:
{
"Id": "Policy1357935677554",
"Statement": [
{
"Sid": "Stmt1357935647218",
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": "arn:aws:s3::: DestinationBucket",
"Principal": {"AWS": "arn:aws:iam::XXXXXXXXXXXX:user/src–iam-user"}
},
{
"Sid": "Stmt1357935676138",
"Action": ["s3:PutObject"],
"Effect": "Allow",
"Resource": "arn:aws:s3::: DestinationBucket/*",
"Principal": {"AWS": "arn:aws:iam::XXXXXXXXXXXX:user/src–iam-user"}
}
]
}
command to be run is s3cmd cp s3://SourceBucket/File1 s3://DestinationBucket/File1
Best Answer
The
mvcommand has no such option to throttle bandwidth. You can see the full options list here:http://docs.aws.amazon.com/cli/latest/reference/s3/mv.html
There are some configuration values you can customize to control S3 transfers commands (cp, sync, mv, and rm).
For example, setting the value of
max_concurrent_requeststo a value lower than 10 (which is the default), will make it less resource intensive. Have a look at the link below and try to adjust the values and see if it helps.See details here: AWS CLI S3 Configuration