Linux – How to pipe the output of uptime/df to curl

As of GNU coreutils 7.5 released in August 2009, sort allows a -h parameter, which allows numeric suffixes of the kind produced by du -h:

du -hs * | sort -h

If you are using a sort that does not support -h, you can install GNU Coreutils. E.g. on an older Mac OS X:

brew install coreutils
du -hs * | gsort -h

From sort manual:

-h, --human-numeric-sort compare human readable numbers (e.g., 2K 1G)

Unfortunately wget cannot parse JavaScript, so spidering such a site is quite difficult.

The good news is, search engines don't generally parse it either, so they are most likely feeding slightly different content to search engines (which is a bad idea for other reasons) so that they can get their pages indexed. They have to feed search engines pages which are reachable without JavaScript if they want to actually be indexed. If this is the case, you can work around it by spoofing Googlebot with wget, such as:

wget --user-agent="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www. google.com/bot.html)" ...

A very few sites will actually check your IP address to see if you're really Googlebot, but this is far less common than it should be.

Another thing to do is to check for the presence of a /sitemap.xml file and use it as a list of URLs to crawl. Some sites provide this file for Google and other search engines to use to spider their content, but nothing says you can't also use it...