Java – Upload to s3 with curl using pre-signed URL (getting 403)

amazon s3bashcurljava

I'm using curl to call into a Java ReST API to retrieve a URL. Java then generates a pre-signed URL for S3 upload using my S3 credentials, and returns that in the ReST reply. Curl takes the URL and uses that for upload to S3, but S3 returns 403 "The request signature we calculated does not match the signature you provided. Check your key and signing method."

Here is the code I'm using to generate the pre-signed URL:

public class S3Util {
    static final AmazonS3 s3 = new AmazonS3Client( new AWSCredentials() {
        @Override
        public String getAWSAccessKeyId() {
            return "XXXXXXX";
        }
        @Override
        public String getAWSSecretKey() {
            return "XXXXXXXXXXXXXX";
        }
    });
    static final String BUCKET = "XXXXXXXXXXXXXXXXXXXXXXXXXXX";

    static public URL getMediaChunkURL( MediaChunk mc, HttpMethod method ) {
        String key = ...
        //way in the future (for testing)...
        Date expiration = new Date( System.currentTimeMillis() + CalendarUtil.ONE_MINUTE_IN_MILLISECONDS*60*1000 );

        GeneratePresignedUrlRequest req = new GeneratePresignedUrlRequest(BUCKET, key, method);
        req.setExpiration(expiration);
        req.addRequestParameter("Content-Type", "application/octet-stream");

        //this gets passed to the end user:
        return s3.generatePresignedUrl(req);
    }
}

and in curl, run from bash, I execute this:

echo Will try to upload chunk to ${location}
curl -i -X POST \
        -F 'Content-Type=application/octet-stream' \
        -F "file=@${fileName}" \
        ${location} || (echo upload chunk failed. ; exit 1 )

Among other things, I have tried PUT, and I have tried "Content-type" (lowercase T). I realize I'm missing something (or somethings) obvious, but after reading the appropriate docs, googling and looking at lots of similar questions I'm not sure what that is. I see lots of hints about required headers, but I thought the resigned URL was supposed to eliminate those needs. Maybe not?

TIA!

Update:

Just to be clear, I have tested downloads, and that works fine.

Java looks like:

GeneratePresignedUrlRequest req = new GeneratePresignedUrlRequest(BUCKET, key, HttpMethod.GET);
req.setExpiration(expiration);

and curl is simply:

curl -i ${location}

Best Answer

I've been able to generate a pre-signed URL via C# and upload it thereafter via curl as expected. Given my tests I suspect you are indeed not using curl correctly - I've been able to upload a file like so:

curl -v --upload-file ${fileName} ${location}

The parameter -v dumps both request and response headers (as well as the SSL handshake) for debugging and illustration purposes:

> PUT [...] HTTP/1.1
> User-Agent: curl/7.21.0 [...]
> Host: [...]
> Accept: */*
> Content-Length: 12
> Expect: 100-continue

Please note, that --upload-file (or -T) facilitates PUTas expected, but adds more headers as appropriate, yielding a proper response in return:

< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< x-amz-id-2: [...]
< x-amz-request-id:  [...]
< Date: Tue, 31 Jan 2012 18:34:56 GMT
< ETag: "253801c0d260f076b0d5db5b62c54824"
< Content-Length: 0
< Server: AmazonS3
Related Topic