I should have bet some cash on this one. It has been pretty obvious for a while.
It seems Rich is taking the right approach as well. Their hard focus on infrastructure is going to pay off. Google’s major competitive advantage to date has been the ability to scale. Up until this point no one has been able to compete due to there massive scale.
That seems to be changing though. The distributed database is starting to take hold.
Sounds like they might need a bigger crawler than Spinn3r though. If they’re going head to head with Google they’re going to need a deep web crawl which is another beast altogether.
I’ve run some numbers but you’re going to need roughly 1-5P of storage for a full copy of the Internet. 1P is about $1M USD (when you factor in redundancy and performance) which is a chunk of change.