Copy s3 contents of bucket, including all versions

amazon s3amazon-web-servicesbackup

I have a few S3 buckets that I want to hand access over to another organisation, as am handing over an existing hosting client to them.

After looking through the docs, if I no longer want to be responsible for the bucket, it seems the simplest option is to copy the contents of the new bucket across to a new bucket controlled by the new organisation, and make any existing apps write files to the new bucket from now on.

The bucket isn't very big:

aws s3 ls --human-readable --recursive --summarize s3://some-client-bucket

 # (snip… lots of files listed, all less than 10mb)

 # Total Objects: 22764
 # Total Size: 2.4 GiB

But some the bucket is versioned, and I have daily snapshots of some files going back the last year which I also want to be able to transfer across.

Is there a straightforward way to do this?

I understand how I might copy the current contents of a bucket to to new one controlled by another organisation after they grant me access using something like:

aws s3 sync s3://some-client-bucket s3://new-client-bucket --recursive

However, I don't think this will move across the versions too, and I'm I've been relying on S3's per file versioning rather than timestamping the files myself.

Do I have to jerry-rig some script to:

  • download each versioned file
  • rename it with a timestamp
  • upload it to the new bucket

Or is there some nifty extra feature in S3 to do this automagically for me?

Best Answer

This one is an old question, but I just happened to come against the same requirement. Today there is a fairly easy way to copy all files from one versioned bucket to another. I came up with the following PowerShell script to do it:

$sourceBucket = <source bucket>
$destinationBucket = <destination bucket>
$nextKeyMarker = $null
$nextVersionIdMarker = $null
do {
    $objectVersions = Get-S3Version -BucketName $sourceBucket `
                                    -KeyMarker $nextKeyMarker `
                                    -VersionIdMarker $nextVersionIdMarker
    $nextKeyMarker = $objectVersions.NextKeyMarker
    $nextVersionIdMarker = $objectVersions.NextVersionIdMarker
    $objectVersions.Versions | foreach { 
        Copy-S3Object -BucketName $sourceBucket `
                      -DestinationBucket $destinationBucket `
                      -Key $_.Key `
                      -DestinationKey $_.Key `
                      -VersionId $_.VersionId `
                      -CannedACLName bucket-owner-full-control
    }
} while ($objectVersions.IsTruncated -eq "True")
  1. This should work no matter the number of objects in the bucket as it copies object versions in bulks of 1000.
  2. The CannedACLName parameter is used to transfer ownership of the objects to the new AWS account where the destination bucket belongs. Without this, the owner of the copied objects will still be the account of the source bucket, making all copied objects inaccessible by the new owner account. If you are copying objects between buckets within the same account, you don't need to specify this.
Related Topic