I have a theory that the hugely complex databases of search engines, like Google and Yahoo, are structured in layers, the deeper layers having more inertia in search results. This would explain the mystery of how my home page URL http://www.asymptoticdesign.co.uk/ appears in search results. At a search using the
site: search operator in Google, or at a search for the site name, the home page URL appears in search results like a URL blocked by the robots.txt file, with only the URL, no title or snippet. While at a search for actual words (just in case some people might do a search for actual words…), like “visualize the World Wide Web as a topological space with sheaves-like structures”, my home page appears nice and bright, with title and snippet. I blocked this URL to search engines for some time with the robots.txt file, while my main site URL was asymptoticdesign.com. Recently I decided to switch, to make asympoticdesign.co.uk my main URL, I redirected HTTP status 301 asymptoticdesign.com to asymptoticdesign.co.uk, and changed the robots.txt file to allow search engines to access and index the new site. I submitted an XML sitemap with the new URL to Google, in Google Webmaster Tools, that was accepted OK, and the stats there show the URL as indexed in search results, and it is indexed for a search query for words included in the content, but not for a search using the
site: operator or for the site name. (The
site: operator restricts the search results to URLs from the specified site.)
I think that the
site: operator search and a site name search are a bit special, and they might access the deeper latent layers of the database, where the “robots.txt-ed out” status has not been updated yet.