I have read about the versioning feature for S3 buckets, but I cannot seem to find if >recovery is possible for files with no modification history. See the AWS docs here on >versioning:
I've just tried this. Yes, you can restore from the original version. When you delete the file it makes a delete marker and you can restore the version before that, i.e: the single, only, revision.
Then, we thought we may just backup the S3 files to Glacier using object lifecycle >management:
But, it seems this will not work for us, as the file object is not copied to Glacier but >moved to Glacier (more accurately it seems it is an object attribute that is changed, but >anyway...).
Glacier is really meant for long term storage, which is very infrequently accessed. It can also get very expensive to retrieve a large portion of your data in one go, as it's not meant for point-in-time restoration of lots of data (percentage wise).
Finally, we thought we would create a new bucket every month to serve as a monthly full >backup, and copy the original bucket's data to the new one on Day 1. Then using something >like duplicity (http://duplicity.nongnu.org/) we would synchronize the backup bucket every >night.
Don't do this, you can only have 100 buckets per account, so in 3 years you'll have taken up a third of your bucket allowance with just backups.
So, I guess there are a couple questions here. First, does S3 versioning allow recovery of >files that were never modified?
Yes
Is there some way to "copy" files from S3 to Glacier that I have missed?
Not that i know of
I was struggling with this, too, but I found an answer over here https://stackoverflow.com/a/17162973/1750869 that helped resolve this issue for me. Reposting answer below.
You don't have to open permissions to everyone. Use the below Bucket policies on source and destination for copying from a bucket in one account to another using an IAM user
Bucket to Copy from – SourceBucket
Bucket to Copy to – DestinationBucket
Source AWS Account ID - XXXX–XXXX-XXXX
Source IAM User - src–iam-user
The below policy means – the IAM user - XXXX–XXXX-XXXX:src–iam-user has s3:ListBucket and s3:GetObject privileges on SourceBucket/* and s3:ListBucket and s3:PutObject privileges on DestinationBucket/*
On the SourceBucket the policy should be like:
{
"Id": "Policy1357935677554",
"Statement": [
{
"Sid": "Stmt1357935647218",
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::SourceBucket",
"Principal": {"AWS": "arn:aws:iam::XXXXXXXXXXXX:user/src–iam-user"}
},
{
"Sid": "Stmt1357935676138",
"Action": ["s3:GetObject"],
"Effect": "Allow",
"Resource": "arn:aws:s3::: SourceBucket/*",
"Principal": {"AWS": "arn:aws:iam::XXXXXXXXXXXX:user/src–iam-user"}
}
]
}
On the DestinationBucket the policy should be:
{
"Id": "Policy1357935677554",
"Statement": [
{
"Sid": "Stmt1357935647218",
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": "arn:aws:s3::: DestinationBucket",
"Principal": {"AWS": "arn:aws:iam::XXXXXXXXXXXX:user/src–iam-user"}
},
{
"Sid": "Stmt1357935676138",
"Action": ["s3:PutObject"],
"Effect": "Allow",
"Resource": "arn:aws:s3::: DestinationBucket/*",
"Principal": {"AWS": "arn:aws:iam::XXXXXXXXXXXX:user/src–iam-user"}
}
]
}
command to be run is s3cmd cp s3://SourceBucket/File1 s3://DestinationBucket/File1
Best Answer
Prior to these changes it was required to create an archive within Glacier and place files within that archive. The link you referenced details how Glacier is now a storage class of S3. You no longer need to move files into Glacier, you can simply upload them as storage class
GLACIER
orDEEP_ARCHIVE
. You can also change the storage type of existing files via the Permission tab or from the command line.From the AWS CLI, you can use a command similar to this:
aws s3 cp /etc/hosts s3://faketest/hosts --storage-class GLACIER
You can see the storage class using
s3api
:aws s3api list-objects --bucket faketest
To do this from the console, click on the Properties tab and select GLACIER
You can similarly set the storage class if you upload a file through the console.
For existing files you can change their storage class through the CLI using something similar to:
aws s3api copy-object --copy-source faketest/temp.txt --bucket faketest --storage-class GLACIER --key temp.txt
The above command copies an existing file from the bucket back to the same bucket with a change to storage class. There may be alternative methods to this.
References
Glacier FAQ
S3 CLI cp
S3 Storage Classes
s3api copy-object