Cuil Hitting Too Hard?

Looks like Cuil might be hitting websites too hard with their crawler:

“I don’t know what spawned it, but when Cuil attempts to index a site, it does so by completely hammering it with traffic,” the tipster wrote. “So much, that it completely brings the site down. We’re 24 hours into this “index” of the site, and I’ve had to restrict traffic to the site down to 2 packets per second, while discarding the rest, or otherwise it makes the site unusable.”

The Admin Zone forums are abuzz over Cuil’s overzealous method for indexing. Countless posters on the site have said that their websites have been brought down because of the Twiceler robot and one user said it “leeched enormous amounts of bandwidth — nearly 2GB this month until it was blocked. It visited nearly 70,000 times!”

One of the benefits to Spinn3r is that we crawl for multiple customers (and we’re super polite).

Cuil might be a bit more aggressive than normal crawlers for now in that they have to backfill archives. Google has the benefit of being around for a while and doesn’t need to drink from the firehose as much as Cuil.



%d bloggers like this: