Linux Split-LRU patch Improves MySQL Swap Performance

For the last few weeks we’ve been running with the Split-LRU patch and 2.6.25.

So far so good…. our boxes don’t seem to page as much anymore and we’re no longer suffering from swap insanity – which essentially fixes the Linux swap problem.

However, it now seems that we’ve been hit by two additional swap related bugs which may or may not be influenced by the Split-LRU patch.

Linux lies about swap utilization:

This is a bit annoying as opposed to a hard stop… When you run free you end up seeing a few megabytes in swap. However, Linux is lying and the data isn’t actually being used.

What I think is happening is that there’s just an accounting bug. Linux is adding pages, then removing them, but not decrementing the counter.

Enhanced load in low memory configurations:

When increasing memory from 87% to 96% we saw radically increased load on the machine. So much so that SSH would not respond and the only solution was to physically reboot the machine.

Bumping the used memory back down to 87% fixed the problem. Of course on 32-128GB machines this is a non-trivial amount of memory to simply waste.

This might be an existing bug which we weren’t able to see before because Linux would swap so fast that this quickly hurt performance.

Any advice on resolving this load issue?

%d bloggers like this: