Facebook can crawl you, but you can’t crawl facebook.

Facebook crawls your like button enabled web pages which can include FBML.

Which is super cool of course.

It’s cool that Facebook is crawling the web.

When does Facebook scrape my Page? “Even if you specify a longer time, Facebook will scrape your page every 24 hours.”

The user agent of the scraper is: “facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php

However, you can’t crawl Facebook. They’re blocking everyone by default (via robots.txt) still.

Unless you’re one of the big boys, you can’t play with Facebook unless you’ve been explicitly whitelisted.

It’s not really the end of the world until you realize that the ENTIRE web would melt if everyone required explicit permission from millions of websites just to crawl the web.

It’s a beg for permission and not a beg for forgiveness model which doesn’t scale.



%d bloggers like this: