Twitter and the Death of Scalability?

Twitter has a post in the Chronicle this afternoon about their recent buzz.

“At this stage, it drives me crazy every day,” Williams said. “On the other hand, if you never have scaling problems, then you spent too much time optimizing before you launched. There’s no fix. It’s always incremental.”

This is a modification of Knuth’s “premature optimization is the root of all evil” but unfortunately they’ve gone the wrong way. Twitter’s law is “premature denial of service is the root of all evil” because this is essentially what is happening.

The Twitter IM bot has been offline for 48 hours. Web pages are taking forever to load, etc.

The clear first step would probably be to deploy Squid. None of their pages can fit in a proxy cache because they’re not setting the right HTTP headers. I’d bet that 80% of Twitter traffic could be offloaded here. Just start here guys.

This is NOT a hard app to scale. It really bother’s me that every new webapp repeats history.

  1. Can’t the first step be deploying something Squid-like? By the time you’re really, really sure you’ve got squid.conf right, you could have had Varnish running and read this month’s Economist. :)

  2. Hey.

    I didn’t think the Squid config was TOO bad. They need to have a minimal config file though……

    At first it’s a bit intimidating.

%d bloggers like this: