Magento – Robots.txt for Magento 2.1+ – where did it go

magento2robots.txtseo

Where is the actual robots.txt file like Magento 1.9.x if it exists?

My site URL /robots.txt does not work.

The official dev docs for this http://docs.magento.com/m2/ce/user_guide/marketing/search-engine-robots.html have an image with a note below it stating

This will be included in before head closing tag in page HTML.

When I inspect the homepage in the tag of the HTML I do not see a reference to the robots.txt or any directives like "INDEX, NOINDEX, FOLLOW, NOFOLLOW". I was searching for more information on this, and think others here probably have as well.

Can someone explain how robots.txt functions in Magento2?

Best Answer

You can find your robots.txt in the folder pub/robots.txt, I recommend you remove it and generate using the admin panel, via the route below you can choose the second option listed and change the SEO robots.txt.

Content > Design > Configuration

Related Solutions

Robots.txt Advice – Need Advice for Optimizing Robots.txt File

According to Google the best, most complete answer to your question is available in Google Webmaster Tools under Crawl > Blocked URLs. Depending on the number of results you have on category pages and the pagination configuration, the

Disallow: /*?

line will block Googlebot from crawling anything but the first page of a category list.

Also, I would not block access to /media/, or at least not to /media/catalog/ or /media/wysiwyg/. Images have value too!

Magento – Magento robots.txt for multi store

I have addressed this issue with a quick little bit of PHP

https://github.com/joekersh/magento-multistore-robots

.htaccess

RewriteRule ^robots.txt$ /robots.php [NC,L]

robots.php

<?php

// Find VirtualHost name and define as variable

$url = $_SERVER['SERVER_NAME'];

// Define multi-store websites url

$website1 = "website1url";
$website2 = "website2url"; 
$website3 = "website3url"; 
$website4 = "website4url";

// Test for requested website then display relevant robots.txt content

if (strpos($url,$website1) !== false) {
    include 'rbts-depend/website1.php';
}

if (strpos($url,$website2) !== false) {
    include 'rbts-depend/website2.php';
}

if (strpos($url,$website3) !== false) {
    include 'rbts-depend/website3.php';
}

if (strpos($url,$website4) !== false) {
    include 'rbts-depend/website4.php';
}



?>

rbts-depend/website1.php

<?php

$content = 
"User-agent: *
# INSERT ROBOTS CONTENT HERE"
;

echo $content;
?>

and so on...