Notes on Spinn3r’s Datacenter Migration to Softlayer

200905251357

About two weeks ago we completed a pretty big project to migrate Spinn3r’s operations from ServerBeach over to Softlayer.

The entire project, from start to finish, too just over one month.

I’m also proud to note that not a single customer noticed any downtime or any issue with our migration. It cost a bit more money and more time but we were able to perform the migration live and in place.

Overview

ServerBeach had been a great dedicated hosting provider for the last two years but we were clearly outgrowing them. Their sweet spot (at least from our perspective) seems to be from the lower end to mid end market. We’re now in the mid to high end dedicated hosting market and Softlayer seems to be the best choice here…

I think my main suggestion to ServerBeach is that they are going to have to come down in price. If they want to compete with Softlayer they’re going to have to do so on price as Softlayer clearly has them beat on feature set.

I did an exhaustive comparison of using cloud providers like Amazon/EC2 vs Colo vs a Dedicated Hosting provider.

My initial thinking was that we’d just throw down $50k-100k and go the colo route. The problem is that the numbers just didn’t add up. It was still far too expensive to go down this route. Internally, there was also a push to look at EC2 and Amazon but they just couldn’t compete on pricing either.

Further, Amazon’s pricing only looks decent if you go down the reserved route. This requires you to put down $30k or so and pay for half of your servers up front. That’s great and all but if you factor in that their servers are a bit underpowered, then you have to buy more of them and your operations costs rise.

I decided to take another look at Softlayer as I evaluated them in mid-2008 and they came in a bit high on pricing. Fortunately, I was able to get an introduction to someone in their Sales department who could appreciate closing a 30-40 server purchase overnight.

Once we had a decent price quote in hand we took a dive into their hardware and pricing model.

I think if it wasn’t for Softlayer we would have had to go down the colo route. IMO they’re by far the best provider in the dedicated hosting space. Serverbeach, Rackspace, etc need to really get their act together if they don’t want Softlayer to eat their lunch.

Further, I have NO idea why these guys aren’t providing Debian. If you want the big Web 2.0 deals you’re going to have to ship Debian. Most of the good Operations Engineers I deal with flat out refuse to work on anything that’s not Debian.

Hardware

The first thing we evaluated was their hardware. They’re using Supermicro 1/2U servers throughout their datacenter (in fact I think they only use Supermicro) which were the same machines we were thinking of purchasing if we went down the colo route.

These are great boxes and I know quite a few large scale shops that have clusters of these machines well into the hundreds of nodes.

Further, their configurations seem to be pretty solid. We can get boxes up to 96MB of RAM and 12TB of storage.

We’re currently using three configurations. API servers are single disk boxes with 4GB of RAM. InnoDB/memory boxes have 32GB of RAM. Our bulk storage and archive boxes have 8GB or RAM, a Adaptec RAID controller and 12TB of storage across 1TBx12 disks.

They also have the Intel X-25 SSDs in stock in which we’ve been interested as well.

200905251403

Network

Their network is pretty impressive as well. They have three datacenters (SEA, WDC, DAL) online now with a 10Gbit interconnect between them. Latency is realistic as well – not as fast as being on gigabit ethernet but certainly reasonable for running replication or serial IO applications over their backend.

Their in-datacenter network uses an all Cisco network with 20Gbit between racks and each machine comes with a 1Gb link. In our benchmarks we were easily able to get full gigabit connectivity to all of our machines. This was something we could not get from Serverbeach.

API and Feature Set

I think the biggest differentiator (and what sold us from my perspective) is the Softlayer API.

There are some pretty interesting features here…

For example, we can shutdown a switch port by calling their API.

If a database is misbehaving, and we need to kill it so that we can do a master promotion, we can just shoot it in the head at the switch port and then promote another master.

This way I can verify that all clients timeout and reconnect to the new master. I can then SSH in via another ethernet port (which has a firewall only allowing SSH) and debug the problem.

We can also check their inventory, reboot a machine, look at billing, change reverse DNS configuration, etc.

This would be a real pain to setup if we were going down the colo route.

Combine that with KVM over IP and you have a winner.


  1. Another, sorry!!!

    “Our bulk storage and archive boxes have 8GB or RAM”

    Perhaps “8GB of RAM”

  2. Nice! SoftLayer’s been good to us these last few years, we too came to them after a stint at ServerBeach. I agree they’re the best provider in their market, and they are both smart and tireless in releasing features (storage options, GSLB, CDN, Xen instances, pervasive API, etc.) that keep them ahead of their competitors.






%d bloggers like this: