Deleting an Amazon AWS S3 bucket with many thousands of files/keys

amazon s3amazon-web-services

Is there an easy way of deleting an Amazon S3 bucket that has many tens of thousands of objects within it? I've tried s3cmd with –recursive and –force, but it memory-bloats and has been running for days with little effect.

Best Answer

The new answer for deleting an Amazon S3 bucket with many thousands or millions of items in it is to use the new Object Expiration rules.

You can even do this from the AWS Console. Simply go to the properties of bucket you want to delete, open the LifeCycle tab and setup a new expiration rule with no prefix and 1 day to expire. Wait one day and the objects will be removed. If you have millions (as I have in my case) you could take a few days before the objects all actually disappear, however, you will be happy to note that you no longer pay for those stored items after the expiration period of 24 hours.

Note: The only limitation on this approach is that you can not set the lifecycle rules if you have versioning enabled.

Object Expiration

Related Solutions

Amazon S3 – How to Get the Size of an Amazon S3 Bucket

The AWS CLI now supports the --query parameter which takes a JMESPath expressions.

This means you can sum the size values given by list-objects using sum(Contents[].Size) and count like length(Contents[]).

This can be be run using the official AWS CLI as below and was introduced in Feb 2014

 aws s3api list-objects --bucket BUCKETNAME --output json --query "[sum(Contents[].Size), length(Contents[])]"

Change Amazon S3 storage class on A LOT of files

The problem you are running into is that the latency of each individual request is high. The solution for this problem on AWS is to parallelize requests to a high degree instead of making requests completely sequentially.

AWS/S3 is set up to handle hundreds and thousands of requests in parallel without any degradation in performance. This will likely require you to do a bit of programming using the API, or you can get some benefit (double, triple, ...) with your existing comfortable UI by making the request several times in different windows for different segments of the S3 objects.

Best Answer

Related Solutions

Amazon S3 – How to Get the Size of an Amazon S3 Bucket

Change Amazon S3 storage class on A LOT of files

Related Topic