Destroying MySQL

Update:

This post hit Reddit and now has 10k views. This post was written for the MySQL community who pretty much already read my blog and know that I support MySQL (we use it in production, document bugs, contribute patches, etc).

While Postgres is a great database, MySQL still has a number of features that Postgres doesn’t have. One good example is the use of multiple storage engines (PBXT, InnoDB, MyISAM, etc).

The entire point of this post was to encourage Sun/MySQL to focus on the quality and stability of their releases.

Everyone I talk to who works for LARGE MySQL installations is complaining about the lack of quality and scalability. These problems need to be fixed not routinely swept under the rug.

Original post:

Let’s say we wanted to destroy MySQL. Basically, make sure no one EVER uses it again.

Further, we want to simultaneously con as many people to fall into the MySQL trap and release products only to have MySQL crash and dump core. (This would have the added benefit of completely destroying the MySQL brand as new developers complain how they migrated to MySQL 5.0 only to have it crash.)

Well, I would perform the following:

– Release MySQL 4.1 with crash bugs. Further, release features like subqueries that are pathologically designed to not actually use any indexes.

– When the community complains, promise to never do it again and that you’ve learned your lesson. Spend the next 6-12 months fixing these bugs.

– Release MySQL 5.0 with crash bugs. Basically, cram as many features as possible and only begrudgingly accept patches.

– When the community complains, promise to never do it again and that you’ve learned your lesson. Spend the next 6-12 months fixing these bugs.

– Further, release TWO versions of MySQL and screw over your original community by keeping the latest and greatest releases from them. Also, screw over your new community by making sure the new releases aren’t tested.

– Spend years talking about the quality of the next release and the massive new (and awesome) feature set.

– Slip your release date by 12 months.

– Allow crash bugs to remain in the new release

– The new features you were talking about? Yeah. They should be beta quality at best. Basically, you want them to try the new features but have them crash when deployed into production.

– Further, new features you’ve been talking about this entire time (row based replication), shouldn’t be enabled by default because they’re unstable.

– When the community complains, promise to never do it again and that you’ve learned your lesson. Spend the next 6-12 months fixing these bugs.

This might be a little harsh but I’m trying to make a point.

I don’t think MySQL deliberately set out to accomplish this goal but they seem to be doing a good job.

MySQL is trying to ship features so they can sell to the market. It’s a catch 22. You’re not going to be able to sell an unstable product.

Further, these are anti-features for my company.

I’m not going to be buying MySQL Enterprise anytime soon. Further, I’m actively discouraging people from purchasing it…

Why? It’s just a bad deal. They’re going to give you a binary which isn’t tested by the community and has limited development. How is this a good idea?

I’m going to be throwing down $20-30k in 2009 on MySQL development but it isn’t going to be given to Sun, MySQL, or Oracle. It’s going to be given to companies like Percona, Open Query, or Proven Scaling that actually care about their customers and release stable fixes to MySQL community.

We’re still on MySQL 4.1. We’re just now migrating to 5.0. Why? Because we depend on performance improvements and fixes present in the community fork of 5.0.

These are not features/patches that are present in official Sun/MySQL binaries.

The Percona and OurDelta builds are good examples. They include VERY important patches that are not present in MySQL 5.1 despite years of development.

About a year ago I needed a partitioning engine. I played with MySQL 5.1’s partitioning engine which looked decent but it crashed when put it into production.

I gave them the benefit of the doubt because it wasn’t GA yet. The bug that caused our crash – yeah, it’s still not fixed.

I waited two months and decided that a partitioning engine was too important to the success of my company to wait around considering MySQL’s poor history.

We now have our own partition engine. It’s in production, reliable, and serving our customers. It’s also stable and has more features than the stock MySQL partitioning engine.

This is a recipe for failure.

I have faith that MySQL can turn the boat around off but they’re going to need to totally halt development for a year and concentrate on stability.

Heck, if you make the right decisions, I might even become a customer!


  1. hi kevin.

    thanks for the points mentioned.

    The delay in release for MySQL 5.1 GA has disheartened me but I’ve waited for it.

    Whats ur experience with partitioning in 5.1 GA ? You planning to release ur partition engine sometime in future ?

    Sincerely appreaciate the fact that you’re planning to spend time/money for releasing patches. (I hope u’ll release them as community patches too …. but thats entirely ur call).

    Thanks again for the article and the finer points.

  2. Hey Ranjeet.

    I haven’t played with MySQL 5.1 partitioning since I decided to implement my own partitioning engine.

    It’s actually not a native MySQL engine but instead part of our larger clustered/distributed database which we run on MySQL.

    I want to Open Source it but it would take about 2-4 months of work to see this happen as it’s pretty tightly integrated into our infrastructure.

    It’s nice though :)

    We’ll definitely be Open Sourcing all the MySQL work we sponsor. I think it will be a requirement…..

    Kevin

  3. nabing

    You hit the nail right on the head. With all the screw-ups leading up to MySQL 5.1, I was inclined to think that they are doing this on purpose so that I would be forced into buying their “Enterprise Edition”. Apparently, this is not the case as the “Enterprise Edition” is even more broken than the Community Edition. At least with the Community Edition, you can find and apply patches to fix most of the broken behavior.

  4. Mats Kindahl

    “Further, new features you’ve been talking about this entire time (row based replication), shouldn’t be enabled by default because they’re unstable.”

    Except that the fact is that it was made the default to not confuse people *upgrading* to 5.1, not because it is unstable. It is the default for new setups since that is the value set in the template files.

  5. Pete

    Over the years I have watched MySQL evolve from a minor worthless wart of a database to a major worthless puss filled wart of a database. Over the same period of time I have watched this industry become increasingly dominated by amateurs and wannabes.

    MySQL is symbolic of everything wrong with contemporary software development. It’s completely useless, favored only by idiots and penniless morons on their latest get rich quick kick.

    To all the people that thing MySQL is a valid option as a database, please get the fuck out this industry NOW.

  6. Kevin,

    Just switch to Postgres.

    MySQL might be open source, but the community was never open.

    Sun has never been able to really work well with Open Communities. Just look at Open Solaris — it does have some really cool technology, but its not built by the community — the priorities is the productized Solaris that they sell — OpenSolaris itself is not usable in production.

    Open Community to me, is more important, and its why I stick around Apache Projects — Open Source by license only is not something I’m interested in supporting.

    -Paul

  7. mark

    The problem here is Sun.

    I see a similar pattern of decline happening with Openoffice.

    I really wonder what the guys of Sun are doing in general. They seem to have a weird attitude to do development.

  8. John

    Paul, never say never… ;)

    Take a look at the Apache Derby community (the basis for Sun’s Java DB), supported by Sun with active contributors. http://db.apache.org/derby/

  9. MySQL follower

    I enjoyed your post. I think it applies a little more context to the monty-bashing posts that have hit MySQL over the few days. When 5.0 hit the streets a lot of people think he made that release decision, and it sucked, and I guess this time round he wants to “clear up any gray areas”.

    To be fair, could you mention the BUG# that you hit with partitioning so we don’t hit it in production?

    I don’t want to put the blame on Sun – because it’s not a problem they created – but it’s also not a problem they can solve. The best releases of OpenOffice.org don’t come from Sun, they come from http://go-oo.org/.

  10. Mark Leith

    Hi Kevin,

    I fail to see how “killing MySQL” would help anybody, even the likes of Percona, OurDelta, or Proven Scaling.

    See, the simple fact is, that whilst these 3 (and Google, I might add, who do a lot of the significant work), do provide some pretty important scalability patches for InnoDB (which is handled by Oracle I might add, perhaps you might focus your wrath on them with regards to this, because it sure isn’t MySQL’s fault that Oracle are not incorporating this stuff), the rest of the stuff they do is adding features for the most part – not fixing bugs (though yes, I admit they have fixed some).

    It is the MySQL team that are doing all of the bug fixing, and the above 3 that simply reap those benefits, whilst adding their new features in to what should be “stable releases”. The bugs they have fixed are a *minute* percentage to what the MySQL team fixes.

    “Killing MySQL” would simply leave the bug fixing in the hands of the above, and the community at large. And I would be willing to bet that they would have no interest what so ever in fixing the majority of them.

    One question – you found a crashing bug in partitioning, but instead of fixing the bug, and sending it back to us – you decided to write a whole new storage engine instead? Why not just fix the bug?

    This idea of “community” around the MySQL project seems seriously warped, to me.

    And these statements about “caring about customers” is starting to really get on my nerves. Go chat to any one support engineer in the MySQL Support organization (of which I am a manager), and tell us we don’t care about our customers and the job we do.. Get real.

    Disclaimer: These are of course my own sentiments, and not my companies.

  11. shawn marano

    I’m always excited to hear about others innovation in the open source world.

    However, my company requires support for one real reason – what happens if you – as you the guy making all these great engines, patches, etc. – fail the “curb” test. You know the one, what happens if the guy with all the knowledge gets hit by bus stepping off the curb? Is there a staff of folks at your company ready to step in with everything you’ve done?

    My company doesn’t have that…though I would like to make some of the extensions, etc. they won’t let me….maybe its’ time to find a place that will.

    shawn

  12. @ Paul Querma: that pg-anti-mysql note is unhelpful; please get over it. You’re quite entitled to your opinion and you may even be “right”, but there’s no reason to keep pestering people about it at ever half-opportunity, even if they are and will remain “wrong”. Really. I wouldn’t want to be part of a community with that kind of attitude. Bad first impression!

    @ Mark: Open Sourcing non-recurring engineering is good for quality, as more eyes can take a peek at it (particularly as a patch), and more people can try it (for instance in an OurDelta -sail bleeding edge build). It’s not an absolute necessity, but I would generally recommend and encourage it.

  13. Ugh – sorry Kevin for calling you Mark….

  14. Kevin Mark

    Agree with Paul. Switch to Postgres. I’m just a user of databases, and not representing any communities.

    Can not believe a single word in Paul’s comment would inflict such strong FIRST impression from Arjen…

  15. VadimTk

    To Mark Leith,

    You seem very sensitive when question about third-party patches is arisen. What do you think is the first-place reason why companies like mine (Percona) need to produce its own patches ?

    Mark, also can you answer me, if you care about customer, will you suggest them to install MySQL with custom scalability patch if you see their system is hurt by scalability problem ?

  16. @Paul Querna

    I agree that open community development is important. I’m hopeful that Drizzle and the community arising around InnoDB will help solve this problem.

    The key point of this post was to point out the flaws with the current MySQL model and hope that Sun fixes it. We’ll see. Either way I think it’s going to be resolved.

    @Mark Leith

    I’m not actually advocating that we should kill MySQL :). This was just a rhetorical device used to point out faults with the current development model.

    While a LOT of the fixes in the Percona build are targeting InnoDB there are a number of fixes in the Google build (specifically targeting replication) that should be fixed in MySQL proper.

    Further, it has been YEARS since Google released these fixes and MySQL did not adopt them.

    “One question – you found a crashing bug in partitioning, but instead of fixing the bug, and sending it back to us – you decided to write a whole new storage engine instead? Why not just fix the bug?”

    Because you don’t except patches. :)

    Further, I wanted new features that weren’t in the original partitioning engine, and it was a TON of code to audit that was a moving target.

    It would have cost a ton more to fix the stock MySQL partitioning engine.

    “Go chat to any one support engineer in the MySQL Support organization (of which I am a manager), and tell us we don’t care about our customers and the job we do.. Get real.”

    I don’t think this is the problem. This ia a high level decision being made by executives at Sun to release a product far too early with far too many features.

    I know for a fact that every MySQL engineer that I’ve met cares a great deal about MySQL. (including the

  17. Jeffrey Pugh

    Kevin, sorry if I missed it in your post, but what is the specific Bug# of the partitioning crash? Although this may not be relevant to you any more, I can certainly provide some feedback on the bug.

    Regarding some of your other points, I’d echo what Mats and Mark have said. We made RBR disabled-by-default in 5.1 for existing users because although it fixes various problems experienced in SBR, it also would concomitantly bring a behavior change to production apps. We’re suggesting instead that users move to RBR as they are ready.

    I don’t think I’d characterize Percona and other’s work as “forks” – the amount of code changed is minimal. And as Mark says, changes to InnoDB are done principally by Innobase. Scalability fixes require a lot of lab and real-world testing, and I certainly hope that Community feedback will lead to Innobase being able to pick the right fixes to apply to InnoDB. For our part, we’re thoroughly testing and benchmarking different patches on a Scalability version of 5.1, and expect to have that available for limited alpha testing in the New Year.

    I’m sorry that your experience with MySQL has made you so negative about it, and can only tell you that we remain committed to the success of all of our customers and users.

  18. @VadimTk

    Vadim, my dear friend,

    Vast majority of your patches are InnoDB related, and we, at Sun / MySQL, don’t have control of InnoDB code, as you know very, very well.

    Regarding scalability patches, yes, we have a number of customers using those patches. We can, however, solve most of the problems without them, as we are so much better experts then you are …. ;o)

    @burtonator

    Most of your comments are truly valid and hold water. Regarding replication patches from Google, they are truly good, but, most of those do affect InnoDB code as well, so we can’t touch them. And, what is more important, we are developing fine new solutions, some of which are popping up in our 6.0 tree.

  19. @Jeffrey Pugh

    Fork or branch would work. I wasn’t really asserting that it was a hard fork.

    Good to hear about the 5.1 scalability patches.

    Just for the record I’m not really negative on MySQL… just trying to give constructive feedback :).

    I *would* certainly like to see the situation improve though.

    @Sinisa

    The global transaction ID stuff is really nice and I don’t think it touches InnoDB.

    These haven’t been isolated and I’d really like to see these land at some point.

    This is probably another area we should fund for a public port of the patch.

    Kevin

  20. Finally someone publicly agreeing with Monty. I think the MySQL development and release model is seriously broken at the moment. With 5.1, they were working an a large pile of features in parallel, they also started a 6.0 branch which has some additions I’m really looking forward to, then there is of course always the enterprise version (and now 2). I think the last 2 major releases (5.0 and 5.1) suffered from featureitis. They should focus on less features but better quality.

    5.0 added many features for the lack of which MySQL has previously been blamed: Views, Stored Functions/Procedures, Triggers and so on. Unfortunately, Views are not really useful because they suffer the same performance problems that subqueries face. As for 4.1, the most significant addition might be Subqueries. I have been a MySQL User since somewhere around 3.21, and most of the time never missed those features. I also used PostgreSQL, and guess what, I didn’t need those there either.

    The largest new Feature in 5.0 is possibly MySQL Cluster, this seems to be work well for those who use it the way it is supposed to be used.

    5.1 delivers something I was really looking forward to, the fully pluggable storage engine interface and the possibility for a lot of new storage engines. From my point of view, that would be enough for a new release. I think they got it pretty much right.

    Row based replication would be the next big thing for me, getting it right and then releasing it as a 5.2 would be great.

    As for partitioning, I don’t use it, it will become interesting if they add more parallelism. Currently I’m partitioning manually (that is creating multiple tables) and that works fine, although things like foreign key constraints don’t really work.

    Partitioning would become interesting if it supports different storage engines per partition, running things like Optimize on single partitions and parallelizing. That would be my personal 5.3.

    Event Scheduler seems to be a less intrusive addition so that could have come along the way. anytime.

    My point is, better take some small steps than try a real big release which was frozen in beta-ish state for 2 years. This also seemed to prevent addition of scalability improvements, and that’s really bugging me, there are scalability patches but come on… At the same time selling boxes which can do 256 threads in parallel and offering a database which doesn’t scale well beyond 4 cores?

    I have great hopes that drizzle gets it right. As for Sun/MySQL, I think it’s pretty much screwed. I don’t know what their strategy or target audience is. An open source project doesn’t need a big new releases with shiny new features, we need consistency and constant improvement. That doesn’t go well with marketing I guess, but after all Sun sells Services and Hardware, not Software.

  21. VadimTk

    Hello Sinisa!

    Of course I read there http://blogs.sun.com/mrbenchmark/entry/scaling_mysql_on_a_256 how you solve scalability problems :)

  22. Another thought,

    If MySQL *really* owned the stability and performance of their product I would have NO problem buying Enterprise.

    Especially if I had to wait 6-12 months between community releases.

    Right now I assume Enterprise is going to be buggy and unstable because the community is doing QA and performance improvements.

    Kevin

  23. Mark Leith

    @Vadim, I’m not sensitive about your patches in particular (I actually think it’s great that you guys have put the time in to this), read my comment again. I also stated the primary reason you have to produce your own patches – primarily because *Oracle*/*InnoDB* are not accepting them. Not MySQL – we have to wait for them to come downstream. Pointing your finger at us as the problem there is futile. We now have microsecond level slow query logging in 5.1 now, don’t we? Where did that come from? ;)

    What I’m sensitive about is the fact that a lot of people now seem to state pretty much all the time that we don’t care about our customers, because we don’t add features in current GA versions. We *do* care about our millions of users, and that’s exactly why we can not accept new features or feature changes in to GA versions.

    And yes, we have been known to point people at the InnoDB scalability patches.

    @Kevin: We *do* accept patches, especially bug fixes, we just can not accept *feature changes* in GA versions. There’s been some silliness with 5.1 and it’s early beta/RC, sure, but the spin that we simply don’t accept things is simply wrong.

    “I don’t think this is the problem. This ia a high level decision being made by executives at Sun to release a product far too early with far too many features.”

    Please keep in mind that when you write what you did in your blog, that you are not just having a go at the management of MySQL, you’re insulting hundreds of engineers that work day in and day out, through their love of MySQL the database, the company, and their cooworkers, to make the product better and better.

    We do love the product. We do care for our customers. We do care about the community.

    It would be nice if the community showed a little love for the unsung heroes behind the scenes that push themselves every day to make MySQL better, in any way they can, as well. And then maybe, just maybe, turn their wrath on the developers of the storage engine that produce the “important patches” everybody is shouting for.

  24. Steve Curry

    Thanks for your post Kevin,

    Based on your update, it seems like you’ve experienced the blessing and curse of a big spike in readership.

    At Sun/MySQL, we really do understand/appreciate/listen to members of the community when they speak passionately about MySQL. Even (maybe *especially*) when they are critical of the product or our model. We constantly try to balance and align our business goals with community interests because they are both important.

    The unfortunate but I guess unavoidable part of open blogging is how it can be mis-interpreted by those from outside the community, or mis-used by those with their own agenda.

    But by all means let’s keep up the dialog — although it may not always seem clear, we really do share the same goal (and personally, I’d like to think we’re getting closer rather than farther apart).

  25. Mark,

    Not adding features to a GA version is a good thing, stability is a requirement and at least for me MySQL 5.0 has been rock solid and working for years.

    As you said, the silliness lies in the early beta stadium of 5.1, so that new feature all go into the 6.0 tree and there is still uncertainity how long it will take to reach GA stage. I’d rather have less features in one release.

    Regarding Oracle, they seem to be finally picking up pace which is strongly related to 5.1 and I hope to see more improvements now that the storage engines can be seperated from the core. As I stated earlier this was the most exciting thing about 5.1 for me as storage engine vendors don’t need to wait for their code to be included in the main product. Just look at what the NDB guys accomplished.

    Also I think there is a line between insult and criticism, I don’t think that Monty (in his blog entry, he definitely doesn’t blame the developers!) and Kevin crossed it.
    This one however crossed the line:
    http://trainedmonkey.com/2008/12/3/an_observer_of_his_own_legacy

    Most of the criticism is directed at the release schedule and priorities and general community policy, not at the quality of the work. So the question is, who is really to blame?

    I am still waiting for comments from Sun/MySQL executives regarding the criticism of 5.1 however.

    Last but not least, who am I to complain? After all it’s free ;)

  26. I personally ran into a brick wall testing MySQL clustering. After having spent 3 or so weeks finally getting a workable configuration, I ran into a giant unrecoverable crash just before deploying to production and it basically proved to be too fundamentally buggy to even consider using it. I filed a MySQL bug report only to hear back 4 months later asking for more log files (as if anyone would hang on to log files after 4 months). Huge disappointment.

  27. VadimTk

    Mark,

    I do respect MySQL Support engineers and do respect MySQL Software engineers, I’ve never had thought to blame them. I can’t recall anyone who did.

    But something wrong is happening, and Kevin is pointing on in this post. I believe this something wrong is coming not from MySQL Support & Development teams.

  28. @VadimTk

    Everyone I’ve met at MySQL from the developers on up to senior management have been nice people and clearly cared a LOT about MySQL.

    The blame doesn’t lie there… there’s just some sort of emergent property happening where they’re continually adding features while risking the reliability of the product.

  29. A number of people have commented about the patches various community members have made to InnoDB, and the fact that so far these changes have not been incorporated into the official InnoDB as developed by Innobase.

    Generally, we at Innobase/Oracle have been watching the community activity with interest. We are familiar with some the proposed patches and have been evaluating them for possible inclusion in InnoDB. To echo Jeffrey’s comments, a great deal of testing is required to validate performance-related changes, as these involve the most critical elements of a database or storage engine.

    We’ve been working closely with Google regarding one such performance-oriented patch, for example. We have identified some issues and subsequently resolved them, but there is still more to do. We haven’t as yet seen the results of Sun/MySQL’s lab testing (though we’d love to!). We also note that the publicly visible testing we’ve seen to date represent different and fairly narrow workloads and often demonstrate performance with more than one independent patch applied. While community feedback is helpful, testing in a scientific manner is essential.

    Several things must be considered in determining whether or not a community contribution can be incorporated into InnoDB. First, we must have complete confidence in the technical correctness and reliability of the code. We want to maintain the reliability of InnoDB, and ensure that the proposed changes have a positive impact on a wide range of workloads. Second, we must believe that the approach taken in a proposed patch is the best way to address a problem or need. We may have plans or an idea for more comprehensive or better architected functionality. Last and not least, since we license InnoDB commercially, we must have complete control over intellectual property. Software contributed under the GPL cannot be licensed commercially, although code covered by the BSD license can be incorporated in a commercially-licensed product. We need complete confidence of the provenance of contributed code.

    For a variety of reasons, we can’t evaluate every patch that appears in the community, and we can’t publicly predict whether or when a particular patch may be incorporated in InnoDB. You should know, however, that we are not opposed to accepting community contributions under the proper circumstances.

  30. Ceesaxp

    As many have said — don’t move to 5.0, move to PostgreSQL. Sure, this will be a difficult transition, but it is well worth it. How does lack of different storage engines affect what you’re doing? MySQL has them simply because the original one was unable to support functionality that was stock in Pg since forever…

  31. If I switch away from MySQL it’s going to be to my own database written in C from the ground up.

    I’ve written nearly the entire stack of Spinn3r other than the kernel and the database so it’s not out of the question :)

  32. @Mark Leith,

    How did micro-slow logging get into 5.1? Rumor is it was rejected, and then Monty decided to subvert that decision and included it as part of something else. If not for Monty’s rebellion, it still wouldn’t be there. That is, if the rumor is true. I haven’t bothered to check the commit history.

    I’m really sympathetic to your comments in this thread. If I were in your place, I believe I’d feel the same way. I’m sorry that the upper-level management’s decisions are causing this backlash that gets directed a little indiscriminately. But seriously, from an outsider’s point of view, nobody says “Sun’s management doesn’t care, although I’m sure there are support engineers behind there somewhere who do.” For better or for worse, from the outside it all looks like one big Sun, and there are no unsung heroes in sight. This is not MY point of view, but it’s certainly what I think the general public sees.

    @Sinisa,

    “Regarding scalability patches, yes, we have a number of customers using those patches. We can, however, solve most of the problems without them,”

    Eh…. tell that to your customers who come to us (Percona) after getting completely frustrated with Sun/MySQL and giving up. It’s not that uncommon for me to answer the phone and hear someone say “I hope you guys can help us. We’ve been working with Sun on this for three weeks and we’ve gotten nowhere. They keep telling us there is no problem.” And then we sometimes need to make a patch to solve it. You should realize that unhappy customers sometimes simply walk away quietly and get help elsewhere.

  33. About 6.0 — it’s not a solution. No one should seriously believe that people have confidence in something being fixed in 6.0 after the delayed release of 5.1. The general sentiment I detect is cynicism that it’s vaporware and could take many years to release. I believe that pointing to “that will be fixed in 6.0” only makes people feel MORE like they’re not being listened to.

  34. I would love to know what workarounds exist. I can use innodb_thread_concurrency=4 and then in some cases an 8+ core server will be as fast as the 4 core server. But I expect the 8+ core server to be much faster.






%d bloggers like this: