Google’s Mobile-Friendly Test Doesn’t Follow Robots.txt Rules

Google’s John Mueller recently advised the company’s mobile-friendly test doesn’t follow the rules written in robots.txt files.

This topic came up in a Google Webmaster Central hangout where a question was asked about the mobile-friendly test and its relation to crawling and indexing.

Here is the question that was submitted:

“If I can see my text in Google’s mobile-friendly test, can I ensure that Google doesn’t have a problem with reading and indexing my content and my pages?”

In response, Mueller said it’s correct that if the mobile-friendly test can render a page then that’s a sign it can be crawled and indexed just fine.

The one thing to watch out for is robots.txt files, which the mobile-friendly test doesn’t respond to. That means it won’t tell you the server response for a page, or whether any JavaScript files are being blocked.

So, in addition to the mobile-friendly test, site owners should be using the inspect URL tool in Search Console to make sure everything is working as intended.

Hear Mueller’s full response in the video below, starting at the 28:36 mark:

“For the most part – yes, that’s correct. There can be subtle issues with JavaScript but, in general, the mobile-friendly test is something that uses the new chromium setup that we use for Googlebot. So if [the web page] works in a mobile-friendly test it should work for indexing as well.

One of the things to watch out for here is the mobile-friendly test – as far as I know – doesn’t follow the robots.txt rules.

So that’s something where you might want to use the Inspect URL tool in Search Console to make sure that, actually, your JavaScript files, and your server responses, and all of that are working well from that point of view.”