Ethernet Latency: The Hidden Performance Killer
The other night I was thinking about MySQL cluster and whether or not a high speed interconnect would be worth the investment. Doing some research lead me to the conclusion that ethernet latency isn’t discussed very often except in niche mailing list threads. I had to do a lot of research to find even a basic discussion of the problem (Note. If you have any good resources on the subject please add a comment.)
In order to help other developers avoid mistakes I’ve made in the past I figure a good blog post on the subject was in order.
What’s ethernet latency? When you send a packet to a remote machine over gigabit ethernet it’s not just instant. There’s overhead involved in sending the packet back and worth. The data has to go from user space into the kernel, out the network interface, onto a switch, and then back up the kernel and into userland of the remote machine (and back again!). All this yields latency.
The speed of a normal ethernet ping/pong (request and response) is roughly 350us (microseconds) or about .35 milliseconds or .00035 seconds.
Of course this sounds blazingly fast but if you need to perform a few thousand operations per second per page within your webapp it can lead to a major (and unforeseen) performance bottleneck. With the above number in hand this yields about 2800 operations per second.
How can you fix this problem? Even if you put everything in memory and use your MySQL query cache and tune the heck out of every component in your system you’re still going to run into the problem of ethernet latency. All you can really do is attempt to batch everything so that you’re only performing a few network operations per page load (even better if you can cache the whole page in squid). See my Transparent Batch and Stream Operations in Distributed Systems for more information on this topic.
What about gigabit ethernet vs 10 gigabit ethernet? Unfortunately the latency is still about the same. You could buy SCI cards which NDB supports but you’re only talking about a 10x performance gain. This yields about 28k operations per second.
The more elegant solution is to batch up your network communication so that all your IO is performed in a few requests. Use bulk INSERT statements and SELECT IN (…) queries within MySQL. In memcached use getMulti() instead of individual gets(). If you’ve developed your application with latency in mind you should be able to get all the data you need to render a page within 2-5 operations. Even if you have to perform 50 operations this will still be within an acceptable range.
One problem I’ve seen with memcached is that it doesn’t have a setMulti method. Thus if you have a cache which is only seeing a 50% hit rate you’re going to be doing a lot of set()s which will slow down your performance significantly. It turns out you can send multiple sets without having to wait for a response so it should be possible to implement setMulti() in the client.
I’m planning on revisiting a few things with the java memcached implementation (including async IO and parallel gets) and a setMulti implementation will certainly be on my short list.