Testing if a server is up with curl and grep. Grep adds headings

curlgrep

Hi I'm trying to see if a server I'm running has been stopped (if it has, it returns a 502 Bad Gateway message.)

So I tried to do this :

curl http://MY_URL | grep "502"

The curl part returns what I expect it to.

But when I pipe it into grep, grep seems to be adding a bunch of headings :

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
113   682  113   682    0     0   230k      0 --:--:-- --:--:-- --:--:--  666k

These aren't in the output from curl. And grep on ordinary text files doesn't add them. So why are they getting added to curl piped through grep?

Best Answer

That output is from curl. It's added when standard output is not a terminal, and in some other circumstances. Use curl -s if you don't want to see it.

Related Solutions

Recursive Text Search with Grep and File Patterns

My version of GNU Grep has a switch for this:

grep -R --include='*.txt' $Pattern

Described as follows:

--include=GLOB
Search only files whose base name matches GLOB (using wildcard matching as described under --exclude).

Difference between `curl -I` and `curl -X HEAD`

It seems the difference has to do with the Content-Length header and how it is treated by both commands.

But before going into that, curl -X HEAD does not give any output because, by default, curl does not print headers if switch -i is not provided (not needed on -I though).

In any case, curl -I is the proper way to fetch the headers. It just ask for the header and close the connection.

On the other hand curl -X HEAD -i will wait for the transmission of the number of bytes stated by Content-Length. In the case no Content-Length is not specified, I guess it will wait for some data or for that particular header.

Some examples that shows this behaviour:

$ curl -X HEAD -i http://www.elpais.es
HTTP/1.1 301 Moved Permanently
Server: AkamaiGHost
Content-Length: 0
Location: http://www.elpais.com/
Date: Wed, 12 May 2010 06:35:57 GMT
Connection: keep-alive

Because Content-Length is 0, in this case both commands behave the same. And the connection is closed afterwards.

$ curl -X HEAD -i http://slashdot.org
HTTP/1.1 200 OK
Server: Apache/1.3.41 (Unix) mod_perl/1.31-rc4
SLASH_LOG_DATA: shtml
X-Powered-By: Slash 2.005001296
X-Bender: Since I love you all so much, I'd like to give everyone hugs.
X-XRDS-Location: http://slashdot.org/slashdot.xrds
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
Content-Length: 115224
Date: Wed, 12 May 2010 06:37:20 GMT
X-Varnish: 1649060825 1649060810
Age: 1
Connection: keep-alive

curl: (18) transfer closed with 115224 bytes remaining to read

In this case, there seems to be a timeout (probably by Varnish), so curl protests that the connection was closed before having received the Content-Length number of bytes.

By the way, look at the funny X-Bender (shown in the example) and X-Fry (try it for yourself) headers :).

Best Answer

Related Solutions

Recursive Text Search with Grep and File Patterns

Difference between `curl -I` and `curl -X HEAD`

Related Topic