.Google.com's John Mueller addressed an inquiry regarding why Google.com indexes pages that are refused from crawling through robots.txt and also why the it is actually safe to disregard the related Search Console records regarding those crawls.Crawler Traffic To Concern Guideline URLs.The person asking the question chronicled that robots were producing links to non-existent concern specification URLs (? q= xyz) to pages with noindex meta tags that are actually likewise shut out in robots.txt. What triggered the concern is that Google is creeping the links to those webpages, getting shut out by robots.txt (without seeing a noindex robotics meta tag) at that point acquiring turned up in Google.com Look Console as "Indexed, though blocked through robots.txt.".The individual asked the observing concern:." Yet right here is actually the big concern: why will Google.com index web pages when they can not also view the web content? What is actually the advantage in that?".Google.com's John Mueller confirmed that if they can not crawl the webpage they can not find the noindex meta tag. He additionally produces an appealing mention of the website: search driver, encouraging to dismiss the results due to the fact that the "average" consumers won't observe those end results.He created:." Yes, you are actually right: if we can not creep the webpage, our experts can not see the noindex. That mentioned, if our team can't crawl the pages, after that there's not a great deal for us to mark. Therefore while you could observe several of those webpages along with a targeted web site:- concern, the average user will not find all of them, so I wouldn't fuss over it. Noindex is actually likewise alright (without robots.txt disallow), it only implies the Links will end up being crept (and also end up in the Explore Console file for crawled/not recorded-- neither of these conditions induce issues to the rest of the website). The integral part is actually that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's solution validates the restrictions in operation the Web site: search advanced hunt operator for diagnostic main reasons. One of those reasons is actually considering that it is actually certainly not connected to the routine hunt index, it is actually a distinct point altogether.Google.com's John Mueller discussed the internet site hunt operator in 2021:." The quick answer is actually that a web site: inquiry is actually certainly not meant to be full, nor used for diagnostics purposes.A web site question is a specific kind of search that limits the end results to a specific site. It's essentially only words internet site, a colon, and after that the site's domain.This question confines the outcomes to a particular website. It's not indicated to be an extensive compilation of all the web pages coming from that web site.".2. Noindex tag without using a robots.txt is actually great for these kinds of scenarios where a bot is linking to non-existent web pages that are obtaining found out by Googlebot.3. URLs along with the noindex tag will definitely produce a "crawled/not recorded" item in Look Console and that those won't have a bad impact on the remainder of the internet site.Read the inquiry and respond to on LinkedIn:.Why would certainly Google.com mark pages when they can not also find the information?Included Photo by Shutterstock/Krakenimages. com.