Deleting multiple objects in a AWS S3 bucket with s3curl.pl

amazon s3amazon-web-services

I have been trying to use the AWS "official" command line tool s3curl.pl to test out the recently announced multi-object delete. Here is what I have done:

First, I tested out the s3curl.pl with a set of credentials without a hitch:

$ s3curl.pl --id=s3 -- http://testbucket-0.s3.amazonaws.com/|xmllint --format -
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100   884    0   884    0     0   4399      0 --:--:-- --:--:-- --:--:--  5703
    <?xml version="1.0" encoding="UTF-8"?>
    <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
      <Name>testbucket-0</Name>
      <Prefix/>
      <Marker/>
      <MaxKeys>1000</MaxKeys>
      <IsTruncated>false</IsTruncated>
      <Contents>
        <Key>file_1</Key>
        <LastModified>2012-03-22T17:08:17.000Z</LastModified>
        <ETag>"ee0e521a76524034aaa5b331842a8b4e"</ETag>
        <Size>400000</Size>
        <Owner>
          <ID>e6d81ea69572270e58d3814ab674df8c8f1fd5d502669633a4951bdd5185f7f4</ID>
          <DisplayName>zackp</DisplayName>
        </Owner>
        <StorageClass>STANDARD</StorageClass>
      </Contents>
      <Contents>
        <Key>file_2</Key>
        <LastModified>2012-03-22T17:08:19.000Z</LastModified>
        <ETag>"6b32cbf8219a59690a9f69ba6ff3f590"</ETag>
        <Size>600000</Size>
        <Owner>
          <ID>e6d81ea69572270e58d3814ab674df8c8f1fd5d502669633a4951bdd5185f7f4</ID>
          <DisplayName>zackp</DisplayName>
        </Owner>
        <StorageClass>STANDARD</StorageClass>
      </Contents>
    </ListBucketResult>

Then, I following the s3curl.pl's usage instructions:

s3curl.pl --help
Usage /usr/local/bin/s3curl.pl --id friendly-name (or AWSAccessKeyId) [options] -- [curl-options] [URL]
 options:
  --key SecretAccessKey       id/key are AWSAcessKeyId and Secret (unsafe)
  --contentType text/plain    set content-type header
  --acl public-read           use a 'canned' ACL (x-amz-acl header)
  --contentMd5 content_md5    add x-amz-content-md5 header
  --put <filename>            PUT request (from the provided local file)
  --post [<filename>]         POST request (optional local file)
  --copySrc bucket/key        Copy from this source key
  --createBucket [<region>]   create-bucket with optional location constraint
  --head                      HEAD request
  --debug                     enable debug logging
 common curl options:
  -H 'x-amz-acl: public-read' another way of using canned ACLs
  -v                          verbose logging

Then, I tried the following, and always got back error. I would appreciated it very much if someone could point out where I made a mistake?

$ s3curl.pl --id=s3 --post multi_delete.xml -- http://testbucket-0.s3.amazonaws.com/?delete
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><StringToSignBytes>50 4f 53 54 0a 0a 0a 54 68 75 2c 20 30 35 20 41 70 72 20 32 30 31 32 20 30 30 3a 35 30 3a 30 38 20 2b 30 30 30 30 0a 2f 7a 65 74 74 61 72 2d 74 2f 3f 64 65 6c 65 74 65</StringToSignBytes><RequestId>707FBE0EB4A571A8</RequestId><HostId>mP3ZwlPTcRqARQZd6gU4UvBrxGBNIVa0VVe5p0rqGmq5hM65RprwcG/qcXe+pmDT</HostId><SignatureProvided>edkNGuugiSFe0ku4eGzkh8kYgHw=</SignatureProvided><StringToSign>POST


Thu, 05 Apr 2012 00:50:08 +0000

The file multi_delete.xml contains the following:

cat multi_delete.xml
<?xml version="1.0" encoding="UTF-8"?>
<Delete>
    <Quiet>true</Quiet>
    <Object>
      <Key>file_1</Key>
      <VersionId> </VersionId>>
    </Object>
    <Object>
      <Key>file_2</Key>
      <VersionId> </VersionId>
    </Object>
</Delete>

Thanks for any help!

–Zack

Best Answer

This worked for me (Amazon Linux AMI v2012.03; us-east-1):

Download and extract latest s3curl.zip

wget http://s3.amazonaws.com/doc/s3-example-code/s3-curl.zip
unzip s3-curl.zip

Make executable:

cd s3-curl
chmod +x s3-curl

Create a credential file (.s3curl):

%awsSecretAccessKeys = (
    # personal account
    personal => {
        id => 'XXXXXXXXXXXXXXXXXXXX',
        key => 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
    },
);

Restrict credentials permissions:

chmod 600 .s3curl

Create POST request body (multidelete.xml):

<?xml version="1.0" encoding="UTF-8"?>
<Delete>
    <Object>
         <Key>file1.txt</Key>
    </Object>
    <Object>
         <Key>file2.txt</Key>
    </Object>
</Delete>

Calculate base64 encoded MD5 sum of POST body:

cat multidelete.xml | openssl dgst -md5 -binary | base64
cD8Q8KTug5P8Hj7oyOW8iQ==

Construct request, and enable verbose display for curl (I have included the command above inline, instead of the MD5 sum itself):

./s3curl.pl --id=personal --post multidelete.xml --contentMd5 `cat multidelete.xml | openssl dgst -md5 -binary | base64` -- -v http://s3.amazonaws.com/BUCKET?delete

Notes:

Trying to use PUT (instead of POST) results in a 405 Method Not Allowed:

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><ResourceType>MULTI_OBJECT_DELETE</ResourceType><Method>PUT</Method><RequestId>XXXXXXXXXXXXXXXX</RequestId><HostId>XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</HostId></Error>

Trying to use POST without the Content-MD5 header results in a 400 Bad Request:

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidRequest</Code><Message>Missing required header for this request: Content-MD5</Message><RequestId>XXXXXXXXXXXXXXXX</RequestId><HostId>XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</HostId></Error>

Using a hexadecimal md5 sum results in a 400 Bad Request:

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidDigest</Code><Message>The Content-MD5 you specified was invalid.</Message><RequestId>XXXXXXXXXXXXXXXX</RequestId><Content-MD5>703f10f0a4ee8393fc1e3ee8c8e5bc89</Content-MD5><HostId>XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</HostId></Error>

A valid request/response looks like:

./s3curl.pl --id=personal --post multidelete.xml --contentMd5 `cat multidelete.xml | openssl dgst -md5 -binary | base64` -- -v http://s3.amazonaws.com/BUCKET?delete
* About to connect() to s3.amazonaws.com port 80 (#0)
*   Trying 72.21.194.31... connected
* Connected to s3.amazonaws.com (72.21.194.31) port 80 (#0)
> POST /BUCKET?delete HTTP/1.1
> User-Agent: curl/7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/3.12.9.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
> Host: s3.amazonaws.com
> Accept: */*
> Date: Thu, 05 Apr 2012 01:50:53 +0000
> Authorization: AWS XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
> Content-MD5: cD8Q8KTug5P8Hj7oyOW8iQ==
> Content-Length: 172
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< x-amz-id-2: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
< x-amz-request-id: XXXXXXXXXXXXXXXX
< Date: Thu, 05 Apr 2012 01:50:55 GMT
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Server: AmazonS3
<
<?xml version="1.0" encoding="UTF-8"?>
* Connection #0 to host s3.amazonaws.com left intact
* Closing connection #0
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Deleted><Key>file1.txt</Key></Deleted><Deleted><Key>file2.txt</Key></Deleted></DeleteResult>

Using <Quiet>true</Quiet> results in the following body returned from a successful deletion:

<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></DeleteResult>

The only way I could replicate your error message was by using an invalid secret key in my credentials file. However, with that the bucket listing did not work either, unlike yours.

Related Topic