Apache Server – How to Create Robots.txt for All Domains

apache-2.2PHProbots.txt

We have a XAMPP Apache development web server setup with virtual hosts and want to stop serps from crawling all our sites. This is easily done with a robots.txt file. However, we'd rather not include a disallow robots.txt in every vhost and then have to remove it when we went live with the site on another server.

Is there a way with an apache config file to rewrite all requests to robots.txt on all vhosts to a single robots.txt file?

If so, could you give me an example? I think it would be something like this:

RewriteEngine On
RewriteRule  .*robots\.txt$         C:\xampp\vhosts\override-robots.txt [L]

Thanks!

Best Answer

Apache mod_alias is designed for this and available from the core Apache system, and can be set in one place with almost no processing overhead, unlike mod_rewrite.

Alias /robots.txt C:/xampp/vhosts/override-robots.txt

With that line in the apache2.conf file, outside all the vhost's, http://example.com/robots.txt - on any website it serves, will output the given file.

Related Solutions

Ssl – Apache: rewrite port 80 and 443 – multiple SSL vhosts setup

Remember that when you make an HTTPS connection to https://domain2.org (which is really a connection to https://domain2.org:443), the order is roughly

Browser opens connection to IP on port 443
Apache looks for a VirtualHost matching the combination IP:443
In this case it finds the VirtualHost for domain1.org
Apache sends the SSL certificate associated with that VirtualHost (i.e. the cert for domain1.org)
[The rest of the SSL negotiation]
Browser sends its HTTP request headers, including the Host: header telling Apache what domain it thinks it's talking to.
If applicable, mod_rewrite processing of the HTTP request headers happens here.

i.e. when Apache decides which certificate to send, it hasn't yet received the information that mod_rewrite would use to do the redirect.

To get around this you'd need a single certificate that was valid for both domains. This could be either a wildcard certificate (if the situation is domain1.example.com and domain2.example.com, get a certificate for *.example.com), or you could try to get an SSL certificate with a Subject of "domain1.org", but a Subject Alternate Name of "domain2.org".

In either case you'd need to do the redirection in the main Apache config files - if it's in a .htaccess file in domain2.org's DocumentRoot, then requests that use domain1.org's VirtualHost will go to a different DocumentRoot directory, and won't ever see the file.

Setting different caching headers for difrerent redirections using Apache mod_rewrite

There is a way to do this, but not with mod_expires. Instead you must set an environment variable using mod_rewrite and then conditionally add the right caching headers with mod_headers as described in Mark S. Kolich: Set the Cache-Control and Expires Headers on a Redirect with mod_rewrite.

So my final solution looks like this:

RewriteRule ... [last,redirect=301,env=longexpiry:1]
RewriteRule ... [last,redirect=302,env=nocache:1]

Header always set Cache-Control "no-store, no-cache, must-revalidate" env=nocache
Header always set Cache-Control "max-age=86400" env=longexpiry

Make sure you don't have a default expiry set by mod_expires though or you will end up with duplicate headers.

Best Answer

Related Solutions

Ssl – Apache: rewrite port 80 and 443 – multiple SSL vhosts setup

Setting different caching headers for difrerent redirections using Apache mod_rewrite

Related Topic