Bash – how to use quoting and escaping with bash

bashcurl

I was given a list of urls that are generating 404 errors as reported to us by google.

I can test a url with curl (from the command line ) like this:

curl -k --user-agent "Googlebot/2.1 (+http://www.google.com/bot.html)" https://MYURLHERE

which works exactly as I expected. I wanted to put this in a script so I can run through a list of them
here is what I have. 

#!/usr/bin/bash
url=$1

curlcmd="curl -k --user-agent \"Googlebot/2.1 (+http://www.google.com/bot.html)\""
$curlcmd $url

but its not working. I keep getting 

curl: (1) Protocol "(+http" not supported or disabled in libcurl

I'm not sure how to escape this to make it work. any suggestions ?

Best Answer

put quotes around your variable $1, or you can use something like this:

$ touch $$
$ echo 'http://www.google.com' >> $$
$ echo 'http://www.yahoo.com' >> $$
$ for url in $(cat $$); do curl -I $url ; done
HTTP/1.1 200 OK
Date: Wed, 22 Nov 2017 15:57:19 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: 1P_JAR=2017-11-22-15; expires=Fri, 22-Dec-2017 15:57:19 GMT; path=/; domain=.google.com
Set-Cookie: NID=117=CaOUCOyr9TPjs64tqyz1MuqHsASzL_3eO5n-NE4ubqAikITGbs7QY0aegNByOWX1Vaf9SsUVQDJ1wdaIOZwXoiqfVZ9ISLtta7tvcDH6LFM52OGFKRH4J5Clde2EX8oG; expires=Thu, 24-May-2018 15:57:19 GMT; path=/; domain=.google.com; HttpOnly
Accept-Ranges: none
Vary: Accept-Encoding
Age: 0
Transfer-Encoding: chunked
Via: 1.1 localhost.localdomain

HTTP/1.1 200 OK
Date: Wed, 22 Nov 2017 15:57:19 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: 1P_JAR=2017-11-22-15; expires=Fri, 22-Dec-2017 15:57:19 GMT; path=/; domain=.google.com
Set-Cookie: NID=117=VRrA0-bCESlSCoerEK0n1hxXfldwpQI4cisiKrEgnKVph9HkfQJu-tbur3ZBiLh3-RFKZ0kbWUWsBwJKzsi_aPUuJzztM1rCuDfljZLxqjaHanZxiCx7qch4P2WCoDDC; expires=Thu, 24-May-2018 15:57:19 GMT; path=/; domain=.google.com; HttpOnly
Accept-Ranges: none
Vary: Accept-Encoding
Age: 0
Transfer-Encoding: chunked
Via: 1.1 localhost.localdomain

HTTP/1.1 200 OK
Date: Wed, 22 Nov 2017 15:57:19 GMT
Via: http/1.1 media-router-fp56.prod.media.ne1.yahoo.com (ApacheTrafficServer [c s f ]), 1.1 localhost.localdomain
Server: ATS
Cache-Control: no-store, no-cache, max-age=0, private
Content-Type: text/html
Content-Language: en
Expires: -1
X-Frame-Options: SAMEORIGIN
Content-Length: 12
Age: 0

$
Related Topic