Sync files from local filesystem with S3 – one way

amazon s3amazon-web-servicess3cmdsynchronization

I have a process that generates log files to the local filesystem, and files that are more than 2 weeks old are automatically purged. I want to use the aws command line tools to sync my folder with S3, with the following behavior:

No sync of files from S3 to local
Files are not deleted on S3 when they are purged on the local filesystem.

How does S3 sync work in this way?

Best Answer

Just use the AWS cli:

$ aws s3 cp --recursive /path/to/local/files/* s3://bucket/

Use a set of IAM credentials that has write-only access to the bucket, so that you can be ensured that no files are ever copied back to the source system.

Related Solutions

Amazon S3 – How to Get the Size of an Amazon S3 Bucket

The AWS CLI now supports the --query parameter which takes a JMESPath expressions.

This means you can sum the size values given by list-objects using sum(Contents[].Size) and count like length(Contents[]).

This can be be run using the official AWS CLI as below and was introduced in Feb 2014

 aws s3api list-objects --bucket BUCKETNAME --output json --query "[sum(Contents[].Size), length(Contents[])]"

Windows – How to delete files older than 30 days with robocopy

I don't know how to do it with robocopy, but when I wanted to do the same as what you're doing, I put this line in my batch file:

forfiles -p %_BACKUPPATH% -m *.rar -d -3 -c "cmd /c del @path"

A dissection is:

forfiles - The program for filtering the files
-p %_BACKUPPATH% - -p indicates "path" (mine is %_BACKUPPATH% but you can replace this with a "real" path), otherwise it uses the current directory
-m *.rar - filter by filetype .rar, skip this parameter for *.*
-d -3 - Only affect files older than 3 days (-3)
-c "cmd /c del @path" - anything after the -c is what will be executed for each file it finds, in this case, del.

Best Answer

Related Solutions

Amazon S3 – How to Get the Size of an Amazon S3 Bucket

Windows – How to delete files older than 30 days with robocopy

Related Topic