Robots.txt Advice – Need Advice for Optimizing Robots.txt File

I have addressed this issue with a quick little bit of PHP

https://github.com/joekersh/magento-multistore-robots

.htaccess

RewriteRule ^robots.txt$ /robots.php [NC,L]

robots.php

<?php

// Find VirtualHost name and define as variable

$url = $_SERVER['SERVER_NAME'];

// Define multi-store websites url

$website1 = "website1url";
$website2 = "website2url"; 
$website3 = "website3url"; 
$website4 = "website4url";

// Test for requested website then display relevant robots.txt content

if (strpos($url,$website1) !== false) {
    include 'rbts-depend/website1.php';
}

if (strpos($url,$website2) !== false) {
    include 'rbts-depend/website2.php';
}

if (strpos($url,$website3) !== false) {
    include 'rbts-depend/website3.php';
}

if (strpos($url,$website4) !== false) {
    include 'rbts-depend/website4.php';
}



?>

rbts-depend/website1.php

<?php

$content = 
"User-agent: *
# INSERT ROBOTS CONTENT HERE"
;

echo $content;
?>

and so on...

To block those kind of URLs with robots.txt you could use the following lines:

Disallow: /*?dir=*
Disallow: /*&dir=*
Disallow: /*?mode=*
Disallow: /*&mode=*
Disallow: /*?order=*
Disallow: /*&order=*
Disallow: /*?p=*
Disallow: /*&p=*

This will tell user-agents not to access all URLs which contain ascending/decending sorting, grid/list mode, position/name/price order and pagination. To learn more about robots.txt files click here.

IMPORTANT: I'm not saying that this is the best way to deal with Magentos sorting and pagination options in relation to duplicate content. There is a lot of discussion about what really is the best way to do this. For instance, you could also use Google Webmaster Tools URLParameters to give Google information about how to handle URLs containing specific parameters. You should be very careful when trying to block URLs because you might unintentionally block the wrong pages.