Caching and Scaling S3

Don posts a great review of his thoughts on S3:

I was going to add a comment but it’s clear that it deserves a dedicated post.

Amazon serves as “cold storage” where everyone’s valuable photos go to live in safety. Our own storage clusters are now “hot storage” for photos that need to be served up fast and furious to the millions of unique visitors we get every day. That’s a bit of an oversimplification of our architecture, as you can imagine, but it’s mostly accurate. The end result is that performance problems with S3 are mostly buffered and offset by our local storage, and even outages are mostly properly handled while resyncing after the outage passes.

I thought about doing this a long time ago and it should be pretty trivial to do.

Off the bat it seems like this would be difficult to setup but actually it’s probably pretty easy.

We’re thinking about using this exact same configuration for Tailrank where we serve images from S3 in a cold store but hot images are served from

This can be easily done with a reverse proxy like Squid.

Further Don goes on to note an important design characteristic for scalable web infrastructure:

It doesn’t matter if you’re GMail or eBay, you have outages and performance problems from time to time. I knew going into this that Amazon would have problems, and I built our software and our own internal architecture to accommodate occasional issues. This is the key to building good internet infrastructures anyway. Assume every piece of your architecture, including Amazon S3, will fail at some point. What will you do? What will your software do?

At Rojo and in Tailrank I designed most functionality to have a failover mode. One solid example is when we add new algorithms or a new feature. If the existing code is working just fine there’s no reason why you can’t write a switch to revert to the old behavior.

A few times at Rojo we had to implement changes to our content store that were a bit risky. I simply added code to revert to the older algorithms so that we could switch back to running the old code at the flick of a switch instead of pushing out a new build.

%d bloggers like this: