Curl and multi-line get parameter

Unfortunately wget cannot parse JavaScript, so spidering such a site is quite difficult.

The good news is, search engines don't generally parse it either, so they are most likely feeding slightly different content to search engines (which is a bad idea for other reasons) so that they can get their pages indexed. They have to feed search engines pages which are reachable without JavaScript if they want to actually be indexed. If this is the case, you can work around it by spoofing Googlebot with wget, such as:

wget --user-agent="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www. google.com/bot.html)" ...

A very few sites will actually check your IP address to see if you're really Googlebot, but this is far less common than it should be.

Another thing to do is to check for the presence of a /sitemap.xml file and use it as a list of URLs to crawl. Some sites provide this file for Google and other search engines to use to spider their content, but nothing says you can't also use it...

As I understand it, curl doesn't follow Location headers by default.

You can enable such behaviour by using the -L or --location switch. Like so:

tom@slappy:~▶ curl -I -L http://shell/redirect.php     
HTTP/1.1 302 Found
Date: Tue, 17 Jan 2012 00:13:54 GMT
Server: Apache/2.2.17 (Ubuntu)
X-Powered-By: PHP/5.3.5-1ubuntu7.2
Location: /target.html#someAnchor
Content-Type: text/html

HTTP/1.1 200 OK
Date: Tue, 17 Jan 2012 00:13:54 GMT
Server: Apache/2.2.17 (Ubuntu)
Last-Modified: Tue, 17 Jan 2012 00:10:37 GMT
Accept-Ranges: bytes
Content-Length: 7
Content-Type: text/html

Note: -I is only used to show the headers rather than page content. I tested this using curl 7.21.6 running on Ubuntu.