We were having some problems with performance a few months ago and my hosting company told me this:
"You have also many hits from all kinds of bots, and when they have too many requests at the website at once it also causes the load to spike."
and
"You should check your website and protect it a bit more in regards to unknown bots."
and
"Regarding the block of search engine crawlers, you can check out the robots.txt file
It is a simple .txt file which is placed in the sites root directory.
To disable all search engine crawlers except google this would be the content (minus the quotation marks)
"User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /"
So I decided to block all bots except for Google – my robots.txt file is located in my Magento root directory and it now looks like this:
User-agent: *
Disallow:
Sitemap: http://www.myw-e-bSITE.com/sitemap/sitemap.xml
User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /
But now I'm thinking maybe this was not such a smart move? (we are not getting as many visitors as before) I mean – should I add Bing and Yahoo also? Any other important Search engines you can think of? And what is the exact code to add Yahoo and Bing?
Best Answer
You should try something like this because what you did is too restrictive.