Using Amazon EC2 and S3 for Cheap Cluster Computing

It looks like Powerset is using Amazon EC2 for their cluster infrastructure. Barney Pell also contributes his thoughts.

Today, Jeff Bezos from Amazon announced the launch of Amazon’s Elastic Compute Cloud (EC2) in Business Week. The launch according to Business Week was met with mixed reviews from Wall Street, negative reviews from analysts, but raving reviews from its customers – like Powerset.

I think the problem here is that analysts wouldn’t understand this strategy. It’s somewhat tangential to Amazon’s core business.

What Amazon should really do is give EC2 away for free to certain startups in exchange for equity. This would make more sense as the interests are slightly more aligned and startups can worry less about Amazon kicking them off their system if they ever get big.

Of course Amazon claims to be scalable but I don’t honestly buy it. Could you build a Google within Amazon’s cluster? Probably not. At some point you’ll have to move away from Amazon if you grow to large but this might be a good problem to have.

This might be the Powerset philosophy. They can bootstrap for cheap and solve the problem sometime in the future. This is essentially the art of bootstrapping a company. Figure out what problems you need to solve now and what problems you should solve once you have more capital.

What I really like about this though is that it provides competition for the existing hosting companies. While I like Serverbeach they need to get their act together fast or I might just switch to EC2.

I’ve been telling them for more than a year now that they need to get gigabit ethernet but they are only now coming around. I suspect that it will be more than another six months.

If they don’t offer a compelling alternative startups are going to bootstrap off of EC2 and a huge percentage of their customer base will just vanish.

Truth be told if I were to run Tailrank over on EC2 I could probably save a bunch of cash. The problem is that it would end up costing me more to migrate right now than anything else. At least two weeks worth of work so I’m not in too much of a hurry.


  1. Dude, I have the same dream as you – to run my site via a scalable solution like EC2. However, EC2 does not persist data, and when the image is torn down or crashes, your data is gone. There’s hints in the forums that they’re maybe building a solution (search for s3fs). Or maybe you’re not talking about web hosting, in which case just ignore me.

  2. > startups can worry less about Amazon kicking them off their system if they ever get big.

    Why would we kick our best customers of of EC2? This makes no sense. The more the merrier, as far as I am concerned.

    > At some point you’ll have to move away from Amazon if you grow to large but this might be a good problem to have.

    It is not at all clear what led you to this conclusion. The economics for each startup are going to be different, but no successful company is ever going to grow out of its need for cheap, scalable processing and large amounts of storage.

    — Jeff;

  3. “Why would we kick our best customers of of EC2? This makes no sense. The more the merrier, as far as I am concerned.”

    A company the size of Google can’t host themselves on Amazon for example. At what point would Amazon feel that one of their customers is using too many compute resources?

    Five thousand machines? One thousand?

    Also…… if you start to rely on local access to S3 and have to leave Amazon due to normal scalability issues the network latencies will start to pile up.

    I think my main point being that the sweet spot is for a startup from from 1-100 servers. At that point it’s probably better to move away from Amazon hosting and do it yourself.

    >> At some point you’ll have to move away
    >> from Amazon if you grow to large but this
    >> might be a good problem to have.

    > It is not at all clear what led you to this
    > conclusion. The economics for each startup
    > are going to be different, but no
    > successful company is ever going to grow
    > out of its need for cheap, scalable
    > processing and large amounts of storage.

    It’s just the normal issues with growing a large company. Google isn’t hosted at one datacenter and having a dependency on a large company isn’t necessarily a good idea.

    Again Amazon seems to be a SWEET alternative for companies for up to 100 servers but at this point you might start to think about doing your own hosting.

    Kevin

  4. I’d love to move my services from ServerBeach to EC2, but the persistent data thing is a biggie for me. While my services deal mostly with transient data, they also have to access databases for user accounts, various logs, etc.

    As I understand it, EC2 is not appropriate for hosting databases. Is this true? Seems like a pretty big functionality gap. If and when the AMZN guys get it addressed, I’m moving.

    Mmmmmm… Elastic.

  5. Kevin:

    My company is in the EC2 beta program. We are using Amazon’s EC2 service (and Amazon’s underlying storage service, S3) to dynamically create and tear down virtual servers to meet fluctuating customer demands and needs. We plan to use EC2/S3 to host hundreds or even thousands of virtual servers, depending on demand. For a small company like Jnana, EC2 and S3 appear to provide a much more cost-effective and practical solution than renting machines in cages from a dedicated hosting service.

    I agree, of course, that EC2 and S3 would probably not make sense for Google or Yahoo or any company running hundreds of thousands or even millions of servers. But to operate several hundred or perhaps even several thousand servers in a dynamic environment, I think EC2 and S3 may be just the ticket. I guess we’ll see soon enough…

  6. ServerBeach launched a gigabit ethernet offering in early December, due in large part to feedback like this from customers like Kevin Burton. (Thanks Kevin)

    The Amazon S3 offering is impressive and challenges ‘traditional’ ways of thinking about hosting in terms of servers, disk space, RAM, etc. Food for thought for hosters like ServerBeach as our offering evolves.

  7. We are using ec2 for deploying our application,its looking good,but expects some more features like data persistence ,multicasting etc.
    Anyone tried dream host and vps http://www.dreamhost.com/hosting-dedicated.html http://www.vpslink.com/vps-hosting/






%d bloggers like this: