Magento – Prevent Search Engine Indexing

categoryindexingproduct

We have number of categories/sub-catgories and products assigned to them in our Magento store.

Is it feasible to make sure that specific category landing pages / product detail pages cannot be indexed by search engines ?

For ex:

http://mysite.com/category_1/sub-category_1.html

is indexed by search engines, while

http://mysite.com/category_2/sub-category_2.html

cannot be indexed by search engines.

The same goes for product details page.

We do not want specific list pages or product detail pages to show up in Google, Bing, etc search results.

How can I achieve this ?

Thanks in advance.

Best Answer

You can use robots.txt to do so. To stop certain category to be indexed by search engine you can use below code in your robots.txt file

User-agent: *
Disallow: http://mysite.com/category_2/sub-category_2.html

You can add the links of whichever category you want in similar fashion. Hope you are clear on this.

To learn more about robots.txt you can refer some links robots.txt, robots.txt tutorial, robots.txt in Magento

To add robots.txt you just need to create a file named robots.txt and upload it on the root of the folder so it is accessible like www.mysite.com/robots.txt

After that you can write the above code in that file and upload it in your root folder

Related Topic