Spinn3r Indexing 52T Per Month

I looked at our bandwidth numbers and Spinn3r has indexed 52T of raw content per month.

That’s 52 TERABYTES people. Nearly 160Mbits continuous IO processed 24/7.

A good portion of this is redundant RSS and polled HTML.

I’d really love to have the web upgraded to support Delta encoding. This would save a ton of money in bandwidth costs.

  1. Yeah… I should have directly mentioned AIM.

    The real issue for us is practical implementation.

    Of course since we crawl for a LOT of customers maybe we could use this to increase usage.


%d bloggers like this: