I am attempting to serve up a different robots.txt for alternate subdomains that I am using for domain sharding so that google doesn't index m1.example.com and m2.example.com.
/etc/httpd/sites-enabled/www.example.com
Alias /robots.txt /var/www/html/robots.txt
....
ServerName www.example.com
ServerAlias m1.example.com m2.example.com
....
DocumentRoot /var/www/www.example.com/public
/var/www/www.example.com/public/.htaccess
RewriteEngine on
RewriteCond %{HTTP_HOST} !=www.example.com [NC]
RewriteRule ^/robots\.txt$ /var/www/html/robots-disallow.txt [L]
....
When I hit m1.example.com/robots.txt it displays the robots.txt not robots-disallow.txt.
If I add the rewrite rule to my vhost in /etc/httpd/sites-enabled/www.example.com it works fine but not in the .htaccess
Best Answer
I solved the problem by removing the forward slash in the .htaccess for ^/robots.txt$ now it looks like:
In the vhost conf it worked with the / in the .htaccess once I removed the / it worked not sure why but that solved it for me.