.Google.com's John Mueller addressed a question about why Google.com indexes webpages that are actually prohibited from crawling through robots.txt and why the it's risk-free to disregard the relevant Explore Console reports regarding those crawls.Robot Visitor Traffic To Concern Parameter URLs.The person asking the inquiry documented that robots were actually generating links to non-existent question criterion URLs (? q= xyz) to web pages along with noindex meta tags that are additionally shut out in robots.txt. What urged the question is that Google is actually creeping the hyperlinks to those pages, obtaining blocked out by robots.txt (without envisioning a noindex robots meta tag) then getting shown up in Google.com Explore Console as "Indexed, though blocked out by robots.txt.".The individual asked the adhering to question:." But right here's the large question: why would certainly Google.com index webpages when they can not also see the web content? What's the conveniences in that?".Google's John Mueller verified that if they can not crawl the webpage they can't find the noindex meta tag. He also makes an interesting reference of the internet site: search driver, encouraging to ignore the results due to the fact that the "common" users won't see those outcomes.He created:." Yes, you're proper: if our team can't crawl the webpage, our company can't observe the noindex. That claimed, if our experts can't crawl the webpages, then there is actually not a whole lot for us to mark. Thus while you could find some of those webpages with a targeted website:- question, the average individual will not find all of them, so I would not fuss over it. Noindex is actually likewise great (without robots.txt disallow), it merely suggests the Links are going to find yourself being crept (and wind up in the Search Console record for crawled/not listed-- neither of these standings trigger problems to the rest of the internet site). The essential part is actually that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's answer confirms the limits in operation the Internet site: hunt progressed search driver for analysis reasons. Among those reasons is actually considering that it is actually certainly not linked to the regular hunt index, it's a separate trait entirely.Google's John Mueller talked about the web site hunt driver in 2021:." The quick answer is that a website: question is actually certainly not implied to become full, neither made use of for diagnostics purposes.A site question is actually a details type of search that confines the end results to a certain web site. It is actually essentially merely words internet site, a colon, and then the site's domain.This question restricts the end results to a certain internet site. It's not suggested to become a comprehensive assortment of all the pages coming from that website.".2. Noindex tag without making use of a robots.txt is actually fine for these type of conditions where a bot is connecting to non-existent web pages that are actually getting found through Googlebot.3. Links along with the noindex tag are going to generate a "crawled/not indexed" item in Search Console and that those won't have a damaging result on the rest of the website.Read the inquiry as well as answer on LinkedIn:.Why would Google.com mark web pages when they can not even observe the information?Featured Picture through Shutterstock/Krakenimages. com.