I implemented S3 multi-part upload, both high level and low level version, based on the sample code from
http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?HLuploadFileJava.html
and
http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?llJavaUploadFile.html
When I uploaded files of size less than 4 GB, the upload processes completed without any problem. When I uploaded a file of size 13 GB, the code started to show IO exception, broken pipes. After retries, it still failed.
Here is the way to repeat the scenario.
Take 1.1.7.1 release,
- create a new bucket in US standard region
- create a large EC2 instance as the client to upload file
- create a file of 13GB in size on the EC2 instance.
- run the sample code on either one of the high-level or low-level API S3 documentation pages from the EC2 instance
- test either one of the three part size: default part size (5 MB) or set the part size to 100,000,000 or 200,000,000 bytes.
So far the problem shows up consistently. I attached here a tcpdump file for you to compare. In there, the host on the S3 side kept resetting the socket.
Best Answer
Although this is c# code, it shows the timeout and partsize settings that I am using to successfully copy files up to 15GB. Perhaps AutoCloseStream needs to be set to false.