Upload large files with curl without RAM cache.

curlhdfshttp

I'm using curl to upload large files (from 5 to 20Gb) to HOOP based on HDFS (Hadoop Cluster) as follows:

curl -f --data-binary "@$file" "$HOOP_HOST$UPLOAD_PATH?user.name=$HOOP_USER&op=create"

But when curl uploading large file it trying to fully cache it in RAM wich produces high memory load. I've tried to use -N flag from man curl which should disable buffering. But nothing happened. So my question is, is there any way to force curl write directly to socket, or could you advice me another util which will cover my needs of simple HTTP POST/PUT requests.

I don't wont to write it by myself since it looks like inventing a bicycle all along. Thank you.

Best Answer

Sadly, according to Daniel Stenberg it's not currently possible.

See https://github.com/bagder/curl/issues/290

Related Solutions

Linux – retrieve file size from HTTP headers without downloading the file with CURL (no HEAD)

Find out why your server does not respond to the HEAD request. This is exactly the purpose of the HEAD request :-)

Another option would be to start fetching the file and then just kill the connection.

Are you calling curl from PHP? In the callback function you could receive the header and then stop.

You can define such a callback with curl_setopt, option CURLOPT_HEADERFUNCTION.

Try using curl_close within this function to stop the transfer. But I did not test this.

Nginx – How to Reply with 200 Without Serving a File

Yes, you can

location / {
    return 200 'gangnam style!';
    # because default content-type is application/octet-stream,
    # browser will offer to "save the file"...
    # if you want to see reply in browser, uncomment next line 
    # add_header Content-Type text/plain;
}

Best Answer

Related Solutions

Linux – retrieve file size from HTTP headers without downloading the file with CURL (no HEAD)

Nginx – How to Reply with 200 Without Serving a File

Related Topic