Google Indexing Technorati Search Results?

I just did a search for “Saddam Execution Video” to research this meme within the search engines.

Check out the 3rd result (bottom of the screenshot):


Niall was quick to point out that this page is blocked by Technorati’s robots.txt

What the heck is going on here?

Maybe this is one way they’re planning on beating Google Blogsearch?

  1. Until recently, their tag pages ranked quite well (T’rati tags was probably one of the most successful SEO initiatives in the history the web). I’m still trying to figure out if their disappearance is because T’rati redesigned the pages or if it was due to Google’s modification of their algorithms towards eliminating sources of duplicate content (they were kind of like an infinite possum, but a little different).

  2. Robots disallow directives mean “don’t visit” — not “don’t return as search result”.

    Why would Google return a result it hasn’t visited? It must have enough information from elsewhere (in-links, toolbar clicktrails, etc.) to determine the page is of likely interest.

    That there’s no ‘snippet’ summary is indicative that they haven’t visited.

  3. One more thought, though I doubt it applies to Google’s bot: their use of the non-standard ‘Crawl-Delay’ directive could give some robots problems.

    I’d put such extensions after all the standard ‘Disallows’, just in case ‘Crawl-Delay’ confuses some robots.

%d bloggers like this: