Amazon S3 upload – bytes transferred is larger than actual file size

The AWS CLI now supports the --query parameter which takes a JMESPath expressions.

This means you can sum the size values given by list-objects using sum(Contents[].Size) and count like length(Contents[]).

This can be be run using the official AWS CLI as below and was introduced in Feb 2014

 aws s3api list-objects --bucket BUCKETNAME --output json --query "[sum(Contents[].Size), length(Contents[])]"

Although this is c# code, it shows the timeout and partsize settings that I am using to successfully copy files up to 15GB. Perhaps AutoCloseStream needs to be set to false.

using (FileStream fileStream = File.OpenRead(file.FullName))
                    {

                        TransferUtilityUploadRequest request = new TransferUtilityUploadRequest()
                        {
                            AutoCloseStream = false,
                            Timeout = 1200000,
                            BucketName = Settings.Bucket,
                            Key = file.Name,
                            InputStream = fileStream,
                            PartSize = 6291456 // 6MB
                        };

                        Console.Write("{0}...", file.Name);
                        Begin();
                        tu.Upload(request);
                        End();
                        Console.WriteLine("Done. [{0}]", Duration());
                    }