S3cmd restore recursive not working right

amazon s3amazon-glaciers3cmd

I am running the following command to restore the data from Amazon S3 to the web server.
s3cmd restore --recursive s3://my-bucket/Weekly-Sunday/dbexports/
and getting this error:

Restore is not allowed, as object's storage class is not GLACIER
On the other hand when I tried –dry-run it didn't show problems.

I am using amazon s3, but not GLACIER and this command to sync:

s3cmd sync LOCAL_DIR s3://my-bucket/Weekly-Sunday/dbexports/ to move data to Amazon s3.

Best Answer

The answer was very simple. The restore option is meant only for Glacier storage: (http://s3tools.org/usage) This little command with inverted parameters would do the job.

s3cmd sync s3://my-bucket/Weekly-Sunday/dbexports/ LOCAL_DIR

Related Solutions

Amazon S3 – How to Get the Size of an Amazon S3 Bucket

The AWS CLI now supports the --query parameter which takes a JMESPath expressions.

This means you can sum the size values given by list-objects using sum(Contents[].Size) and count like length(Contents[]).

This can be be run using the official AWS CLI as below and was introduced in Feb 2014

 aws s3api list-objects --bucket BUCKETNAME --output json --query "[sum(Contents[].Size), length(Contents[])]"

Change Amazon S3 storage class on A LOT of files

The problem you are running into is that the latency of each individual request is high. The solution for this problem on AWS is to parallelize requests to a high degree instead of making requests completely sequentially.

AWS/S3 is set up to handle hundreds and thousands of requests in parallel without any degradation in performance. This will likely require you to do a bit of programming using the API, or you can get some benefit (double, triple, ...) with your existing comfortable UI by making the request several times in different windows for different segments of the S3 objects.

Best Answer

Related Solutions

Amazon S3 – How to Get the Size of an Amazon S3 Bucket

Change Amazon S3 storage class on A LOT of files

Related Topic