Magento – Physical location of robots.txt file in Magento 2.2

magento2magento2.2robots.txt

Where can I find the location of robots.txt file in the Magento root?

Checked pub folder also.

Any one helps.

Best Answer

You have to configured a magento for generating robot.txt file.

Please follow this tutorial

https://support.hypernode.com/knowledgebase/create-robots-txt-magento-2/

Related Solutions

Magento – How to correctly setup robots.txt to block out bad bots but allow Google, Yahoo, Bing and other good bots to access and index the site

You should try something like this because what you did is too restrictive.

User-agent: Yandex
Disallow: /

User-agent: baiduspider
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-Agent: *

Fix Robots.txt and Sitemap.xml Creation Path in Magento 2

This seems to be fixed in Magento 2.2.1, see https://github.com/magento/magento2/pull/9094.

Best Answer

Related Solutions

Magento – How to correctly setup robots.txt to block out bad bots but allow Google, Yahoo, Bing and other good bots to access and index the site

Fix Robots.txt and Sitemap.xml Creation Path in Magento 2

Related Topic