Magento 1.9 – Fix High CPU Load Due to Google Crawling Search Terms

catalogsearchmagento-1.9robots.txt

I'm experiencing a much higher than normal CPU load on my server since a few days mainly caused by excessive search term crawling by Google on my internal Magento search engine. This almost immediately started after I installed the "Sphinx Search" extension. I get the following kind of messages from my server:

1-0 8453 0/0/837 W 9.11 218 0 0.0 0.00 5.33 66.249.78.24 www.mywebshop.com GET /catalogsearch/result/?q=+searchterm+searchterm1+searchterm2 2-0

I've tried several solutions involving the robots.txt file all without any positive result on CPU load reduction. I could always include the following in my robots.txt:

Allowable Index

Allow: /*?q=

In order to stop Google crawlers to abuse my search engine but I don't know whether this is the right way to do it and possible consequences SEO wise. Someone who has dealt with this before and can provide some advise?

Thanks!

Best Answer

All very good suggestions and we have come up with a solution that works very well. The challenge is to tell Google not to do something without upsetting its algorithm to much. Basically we want to slow down Google without being penalized in the end. Good points about the search functionality on the site, this is for customers and definitely not for bots. Eventually this might get you run into trouble since the threat of duplicate content waiting.

What we basically did was tell Google not to use the search functionality by adding the following code to the robots.txt file:

# Allowable Index
Disallow: /catalogsearch/
Disallow: /*?*
Related Topic