How to serve robots.txt for all the own subdomains but not other hosts on Apache

apache-2.2apache-2.4robots.txtvirtualhost

We develop websites and we host the QA environment on the same server as the production environment. I want to serve a specific robots.txt for all QA sites but not for the production sites. We have a lot of sites so I do not want to manyallu update hundreds of vhost configuration blocks. The QA sites are easily identifyable from both the domain name and the directory they are in. QA and production sites are grouped in two different directories. All QA sites are hosted under *.qa.mycompany.com.

Example production:
host: example.org
docroot: /var/www/production/example.org

Example QA:
host: example.qa.mycompany.com
docroot: /var/www/qa/example.org

Is there any way to configure Apache to serve a robots.txt for all QA sites but not the production sites without having to update all QA vhost configs?

Best Answer

You could add this

<Directory "/var/www/qa/*">
  Redirect permanent robots.txt http://example.com/qa_robots.txt
</Directory>

Than you could put your qa-robots.txt on a public areay on a global host, the Directory-block should to a redirect to it for all /qa/ folders.

Search engines should also follow this redirection.

Instead of redirection you can also add

ErrorDocument 404 "/var/www/qa_robots.txt"

But this will send an 404 errorcode I´m not sure how searchbots reackt on this. I think there could also be a way to do this with mod_rewrite and a matching RewriteCond

Related Topic