Spinn3r Indexing 52T Per Month
I looked at our bandwidth numbers and Spinn3r has indexed 52T of raw content per month.
That’s 52 TERABYTES people. Nearly 160Mbits continuous IO processed 24/7.
A good portion of this is redundant RSS and polled HTML.
I’d really love to have the web upgraded to support Delta encoding. This would save a ton of money in bandwidth costs.