Magento – Magento 2.3.0 robots.txt file location in server directory

magento2.3robots.txtseo

I am trying to delete a disallow condition from robots.txt file. This is not located in root or /pub directories.

I can't change it from admin panel because seems like Magento 2.3.0 has a bug on theme edit page.

Github thread for details

Thanks

Best Answer

It seems your Nginx or Apache is set differently. Try locating the file in var directory. If you have SSH access, run below command to see if there is a robots file on server.

Check if there is a robots file on server locate -c robots.txt

If the result found is greater than 0, it means the file exists on the server. Next, you can run below command to find the location

locate robots.txt

Related Solutions

Robots.txt Advice – Need Advice for Optimizing Robots.txt File

According to Google the best, most complete answer to your question is available in Google Webmaster Tools under Crawl > Blocked URLs. Depending on the number of results you have on category pages and the pagination configuration, the

Disallow: /*?

line will block Googlebot from crawling anything but the first page of a category list.

Also, I would not block access to /media/, or at least not to /media/catalog/ or /media/wysiwyg/. Images have value too!

Magento – Magento robots.txt for multi store

I have addressed this issue with a quick little bit of PHP

https://github.com/joekersh/magento-multistore-robots

.htaccess

RewriteRule ^robots.txt$ /robots.php [NC,L]

robots.php

<?php

// Find VirtualHost name and define as variable

$url = $_SERVER['SERVER_NAME'];

// Define multi-store websites url

$website1 = "website1url";
$website2 = "website2url"; 
$website3 = "website3url"; 
$website4 = "website4url";

// Test for requested website then display relevant robots.txt content

if (strpos($url,$website1) !== false) {
    include 'rbts-depend/website1.php';
}

if (strpos($url,$website2) !== false) {
    include 'rbts-depend/website2.php';
}

if (strpos($url,$website3) !== false) {
    include 'rbts-depend/website3.php';
}

if (strpos($url,$website4) !== false) {
    include 'rbts-depend/website4.php';
}



?>

rbts-depend/website1.php

<?php

$content = 
"User-agent: *
# INSERT ROBOTS CONTENT HERE"
;

echo $content;
?>

and so on...