The AWS CLI now supports the --query parameter which takes a JMESPath expressions.
This means you can sum the size values given by list-objects using sum(Contents[].Size) and count like length(Contents[]).
This can be be run using the official AWS CLI as below and was introduced in Feb 2014
aws s3api list-objects --bucket BUCKETNAME --output json --query "[sum(Contents[].Size), length(Contents[])]"
I was struggling with this, too, but I found an answer over here https://stackoverflow.com/a/17162973/1750869 that helped resolve this issue for me. Reposting answer below.
You don't have to open permissions to everyone. Use the below Bucket policies on source and destination for copying from a bucket in one account to another using an IAM user
Bucket to Copy from – SourceBucket
Bucket to Copy to – DestinationBucket
Source AWS Account ID - XXXX–XXXX-XXXX
Source IAM User - src–iam-user
The below policy means – the IAM user - XXXX–XXXX-XXXX:src–iam-user has s3:ListBucket and s3:GetObject privileges on SourceBucket/* and s3:ListBucket and s3:PutObject privileges on DestinationBucket/*
On the SourceBucket the policy should be like:
{
"Id": "Policy1357935677554",
"Statement": [
{
"Sid": "Stmt1357935647218",
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::SourceBucket",
"Principal": {"AWS": "arn:aws:iam::XXXXXXXXXXXX:user/src–iam-user"}
},
{
"Sid": "Stmt1357935676138",
"Action": ["s3:GetObject"],
"Effect": "Allow",
"Resource": "arn:aws:s3::: SourceBucket/*",
"Principal": {"AWS": "arn:aws:iam::XXXXXXXXXXXX:user/src–iam-user"}
}
]
}
On the DestinationBucket the policy should be:
{
"Id": "Policy1357935677554",
"Statement": [
{
"Sid": "Stmt1357935647218",
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": "arn:aws:s3::: DestinationBucket",
"Principal": {"AWS": "arn:aws:iam::XXXXXXXXXXXX:user/src–iam-user"}
},
{
"Sid": "Stmt1357935676138",
"Action": ["s3:PutObject"],
"Effect": "Allow",
"Resource": "arn:aws:s3::: DestinationBucket/*",
"Principal": {"AWS": "arn:aws:iam::XXXXXXXXXXXX:user/src–iam-user"}
}
]
}
command to be run is s3cmd cp s3://SourceBucket/File1 s3://DestinationBucket/File1
Best Answer
That policy works for me. It is likely that you either have a typo in the bucket name, or the policy isn't attached to the user you're using, you're accessing the bucket using a different user than you think you are, etc.