For more on robots.txt, please read robots.txt: the ultimate guide. Or, find the best practices for handling robots.txt in WordPress.
There is a difference between being indexed and being listed in Google
Before we explain things any further, we need to go over some terms here first:
- Indexed / Indexing
The process of downloading a site or a page’s content to the server of the search engine, thereby adding it to its “index.”
- Ranking / Listing / Showing
Showing a site in the search result pages (aka SERPs).
So, while the most common process goes from Indexing to Listing, a site doesn’t have to be indexed to be listed. If a link points to a page, domain, or wherever, Google follows that link. If the robots.txt on that domain prevents indexing of that page by a search engine, it’ll still show the URL in the results if it can gather from other variables that it might be worth looking at.
In the old days, that could have been DMOZ or the Yahoo directory, but I can imagine Google using, for instance, your My Business details these days or the old data from these projects. More sites summarize your website, right.
Now if the explanation above doesn’t make sense, have a look at this video explanation by ex-Googler Matt Cutts from 2009: