A practical way to mirror an Amazon S3 bucket

amazon s3backupcloudmirroringrackspace-cloud

I want to mirror my Amazon S3 buckets. I want to do this because 1) I don't want all my data only existing with one provider; and 2) in case of software error or security breach I want to have the data backed up.

I can mirror to a local disk with the s3cmd sync function, but that does not scale for very large buckets and is not useful for quick backup restores. I'd rather have my data mirrored to a competitor like Rackspace Cloud Files.

Anyone have some suggestions on a simple and robust way to facilitate this kind of mirroring in an automated way on a Linux box?

Best Answer

You can use the "s3cmd" utility with the "sync" option, although I stumbled on your question because I'm trying to figure out if this syncing mechanism is screwing up my duplicity backups.

Related Solutions

Amazon S3 – How to Get the Size of an Amazon S3 Bucket

The AWS CLI now supports the --query parameter which takes a JMESPath expressions.

This means you can sum the size values given by list-objects using sum(Contents[].Size) and count like length(Contents[]).

This can be be run using the official AWS CLI as below and was introduced in Feb 2014

 aws s3api list-objects --bucket BUCKETNAME --output json --query "[sum(Contents[].Size), length(Contents[])]"

Sync/mirror directory with Rackspace Cloud files bucket

The easiest way to sync a local directory to Rackspace cloud files is through the console tools provided by the openstack/swift project. On ubuntu, the tools can be installed with apt-get install python-swiftclient

Then, assuming you are in the directory you want to upload, run the following command in the terminal:

$ swift -A https://auth.api.rackspacecloud.com/v1.0 -U <username> -K <api-key> upload <containername> . --changed

This will recursively upload the files from your current directory to the <containername> container, saving time by uploading only changed files. You need to supply the <username> you use to log in to the Cloud Control Panel and the <api-key> available under Account / Account Settings in the same control panel.

Attention: If you use relative or absolute paths, swift will upload them with the pseudo-path provided on the command line into the container. So if instead of syncing . you sync /var/www/test, then files will end up under /var/www/test pseudo-path of the container - most likely, this is not what you want.

Best Answer

Related Solutions

Amazon S3 – How to Get the Size of an Amazon S3 Bucket

Sync/mirror directory with Rackspace Cloud files bucket

Related Topic