Does CURL cache requests

abcachecdncurlubuntu-14.04

This is a pretty long question, so bear with me.

I wanted to stress my Akamai Server logged in from an AWS instance. So, I started running ab benchmark. However, they seemed ridiculously fast to download ~3 MB video files. Naturally I wanted to see what's going on. This is what I did to get the file

curl -v -o /dev/null

The above completed in ~5 seconds.

Next, I ran the same command again. This time, it completed in ~200ms! Naturally, my intuition says the file is being cached somewhere.

My questions:

Does curl cache files? If so, is there a way to ignore it?
If curl doesn't, does the ubuntu abstracts a cache beneath curl? If so, is there a way to ignore it?
Given the requirements, do you think there could be a benchmarking tool apart from ab that could serve the purpose?

Thank you,
Akshay

Best Answer

The curl client isn't caching files, but the remote server network might well be. Try adding an arbitrary query string variable to the URL to see if you can reproduce it.

Related Solutions

Difference between `curl -I` and `curl -X HEAD`

It seems the difference has to do with the Content-Length header and how it is treated by both commands.

But before going into that, curl -X HEAD does not give any output because, by default, curl does not print headers if switch -i is not provided (not needed on -I though).

In any case, curl -I is the proper way to fetch the headers. It just ask for the header and close the connection.

On the other hand curl -X HEAD -i will wait for the transmission of the number of bytes stated by Content-Length. In the case no Content-Length is not specified, I guess it will wait for some data or for that particular header.

Some examples that shows this behaviour:

$ curl -X HEAD -i http://www.elpais.es
HTTP/1.1 301 Moved Permanently
Server: AkamaiGHost
Content-Length: 0
Location: http://www.elpais.com/
Date: Wed, 12 May 2010 06:35:57 GMT
Connection: keep-alive

Because Content-Length is 0, in this case both commands behave the same. And the connection is closed afterwards.

$ curl -X HEAD -i http://slashdot.org
HTTP/1.1 200 OK
Server: Apache/1.3.41 (Unix) mod_perl/1.31-rc4
SLASH_LOG_DATA: shtml
X-Powered-By: Slash 2.005001296
X-Bender: Since I love you all so much, I'd like to give everyone hugs.
X-XRDS-Location: http://slashdot.org/slashdot.xrds
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
Content-Length: 115224
Date: Wed, 12 May 2010 06:37:20 GMT
X-Varnish: 1649060825 1649060810
Age: 1
Connection: keep-alive

curl: (18) transfer closed with 115224 bytes remaining to read

In this case, there seems to be a timeout (probably by Varnish), so curl protests that the connection was closed before having received the Content-Length number of bytes.

By the way, look at the funny X-Bender (shown in the example) and X-Fry (try it for yourself) headers :).

Execute curl requests in parallel in bash

Use '&' after a command to background a process, and 'wait' to wait for them to finish. Use '()' around the commands if you need to create a sub-shell.

#!/bin/bash

curl -s -o foo http://example.com/file1 && echo "done1" &
curl -s -o bar http://example.com/file2 && echo "done2" & 
curl -s -o baz http://example.com/file3 && echo "done3" &

wait

Best Answer

Related Solutions

Difference between `curl -I` and `curl -X HEAD`

Execute curl requests in parallel in bash

Related Topic